- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considers the non-stationarity and relative overgeneralization problems in cooperative multi-agent reinforcement learning. This paper proposes a fully decentralized framework that infers a discrete latent context from each agent’s sliding-window local trajectory, which aims to capture the teammate-induced environmental dynamics. Empirical validations are conducted on matrix games, SMAC, and a continuous differential game benchmark.
Critical strengths
This paper clearly identifies and formalizes the two core challenges, non-stationarity and relative overgeneralization, in cooperative MARL. It proposes to tackle both challenges by inferring a latent context that captures teammate-induced environment dynamics. The authors offer principled theoretical motivation by proving that the inferred latent context serves as a sufficient statistic of teammates’ joint actions under its assumptions.
Critical weaknesses
-
The discrete context design relies on the claim that dynamics equivalence classes form a discrete set. This assumption is not justified for continuous teammate action spaces. The limited representational capacity of discrete context may restrict the method’s applicability.
-
It is not clearly explained why context inferred from an agent’s recent local trajectory contains relevant information about teammates’ joint actions. In addition, Theorem 1 is built upon the assumption of a deterministic environment.
Questions for authors
-
I remain confused about the definition and design of context. Why is context designed as discrete categories? Is this design motivated by modeling agent actions?
-
Does the method still work if agents only have local partial observations and cannot access the global state?
5 comments
Rebuttal (Part 2)
For clarity, given , we denote the context by and the equivalence class of the other agents' joint action by . Here, denotes the same context as in the main text; this notational change applies only to the rebuttal.
Weakness 1: The discrete context design relies on the claim that dynamics equivalence classes form a discrete set. This assumption is not justified for continuous teammate action spaces. The limited representational capacity of discrete context may restrict the method’s applicability.
1. Response to Weakness 1.
As described in point (4) of the workflow, DAC adopts a finite discrete context space to compute the optimistic marginal value efficiently through enumeration. The discrete design serves this computational purpose; it does not imply that dynamics-equivalence classes are necessarily finite.
The exact correspondence in point (3) requires sufficient context capacity, together with the stated sufficiency and minimality assumptions. For a fixed , continuous teammate actions may induce infinitely many distinct outcomes and hence equivalence classes. In this case, a finite context vocabulary cannot preserve the exact correspondence and can only provide an approximate representation. We will clarify this capacity limitation and distinguish it from the computational motivation for choosing discrete contexts.
A possible extension to continuous contexts involves two components:
(1) Learn a bounded continuous context from local transition windows using a VAE-style objective that reconstructs rewards and next states.
(2) Learn a context-search network to approximately maximize , giving
\approx Q^i\\!\left(s_t,a_t^i,\pi_c^i(s_t,a_t^i)\right).$$ The search network could be optimized through gradients of the value function, following the deterministic actor-update principle used in Deep Deterministic Policy Gradient (DDPG). This would replace finite enumeration with learned context search. It would also require ensuring that proposed contexts are supported for the corresponding $(s_t,a_t^i)$, since bounding the latent space alone does not establish their validity. We leave the development and evaluation of this extension to future work. **Weakness 2:** It is not clearly explained why context inferred from an agent’s recent local trajectory contains relevant information about teammates’ joint actions. In addition, Theorem 1 is built upon the assumption of a deterministic environment. **2. Response to Weakness 2.** **2.1 Information about teammates’ joint actions in local trajectories.** As explained in points (2)–(3) of the workflow, teammates’ joint actions affect the reward and next state observed by agent $i$. Although $a_t^{-i}$ is unavailable, its effects appear in the local transition $(s_t,a_t^i,r_t,s_{t+1})$. The encoder infers $z_t^i$ from this transition, and reconstruction trains it to retain those effects. Under the stated assumptions, $$z_t^i\leftrightarrow(s_{t+1},r_t)\leftrightarrow c_t, \quad \text{for each fixed }(s_t,a_t^i).$$ Thus, information about teammates’ actions is obtained through their observable consequences. The context identifies their dynamics-equivalence class, which may contain multiple joint actions; recovering the exact action is unnecessary. **2.2 The role of deterministic dynamics.** Determinism supports the outcome-class correspondence in point (3). Given $(s_t,a_t^i,a_t^{-i})$, the reward and next state are fixed. In stochastic environments, equivalence classes can instead be defined by their full conditional outcome distributions. One class may then generate multiple outcomes, and different classes may generate the same outcome. When these distributions overlap, a specific local transition $(s_t,a_t^i,r_t,s_{t+1})$ can be consistent with multiple joint-action equivalence classes. Therefore, observing this transition does not, in general, uniquely determine its generating class $c_t$. Even if $z_t^i$ perfectly reconstructs the observed outcome and satisfies Assumptions 2 and 3, it cannot resolve this ambiguity: $$z_t^i\leftrightarrow(s_{t+1},r_t) \quad\not\Rightarrow\quad z_t^i\leftrightarrow c_t, \qquad \text{for each fixed }(s_t,a_t^i).$$ Thus, the one-to-one context-equivalence-class correspondence is not generally guaranteed in stochastic environments. The deterministic assumption in Theorem 1 allows a specific observed outcome to identify exactly one joint-action equivalence class, completing the correspondence established in the workflow.Rebuttal (Part 3)
Question 1: I remain confused about the definition and design of context. Why is context designed as discrete categories? Is this design motivated by modeling agent actions?
3. Response to Question 1.
As explained in point (2) of the workflow, a context is a latent representation inferred from agent ’s observed local transitions and trained through reward and next-state reconstruction. For , the encoder takes , and the decoder reconstructs from . Context categories are learned through this reconstruction objective, without observing or using labels for teammates’ actions.
Following point (4) of the workflow, we choose discrete categories to make optimistic action selection computationally tractable:
A finite context space allows this maximum to be computed by enumeration. The discrete design is therefore motivated by this computation, rather than by assigning a category to each teammate joint action.
The connection to teammates’ actions follows from their effects on rewards and next states. Under the stated theoretical assumptions,
\qquad \text{for each fixed }(s_t,a_t^i).$$ Thus, a context identifies a dynamics-equivalence class that may contain several joint actions. Once the context is inferred, DAC can perform context-based value function learning without recovering the other agents’ exact joint actions. **Question 2:** Does the method still work if agents only have local partial observations and cannot access the global state? **4. Response to Question 2.** The current theoretical analysis does not cover general partially observable settings. As stated in points (2)-(3) of the workflow, the correspondence $$z_t^i\leftrightarrow(s_{t+1},r_t)\leftrightarrow c_t$$ is established for each fixed full state–action pair $(s_t,a_t^i)$. When only local observations are available, context inference instead uses $(o_t^i,a_t^i,r_t,o_{t+1}^i)$. The same $(o_t^i,a_t^i)$ may correspond to different hidden states. Furthermore, distinct global next states associated with different joint-action equivalence classes may produce the same local observation and reward. Consequently, a specific observed transition cannot generally identify the underlying class $c_t$. Even if the context–observed-outcome correspondence holds, $$z_t^i\leftrightarrow(o_{t+1}^i,r_t) \quad\not\Rightarrow\quad z_t^i\leftrightarrow c_t.$$ Partial observability also affects the value-consistency argument: identical observed outcomes may conceal different global next states with different values. Thus, reconstructing $(o_{t+1}^i,r_t)$ does not ensure that the context contains the information required to represent the corresponding joint action value.
Rebuttal (Part 1)
We apologize for our delayed response, as an unexpected power outage in our server room disrupted our additional experiments.
We begin by revisiting the main workflow underlying our work in an accessible manner. For clarity, uppercase letters denote random variables, and lowercase letters denote their realizations.
(1) The generation of per-agent local dynamics. For each agent i, its locally perceived transition and reward distribution is Pπ−ii(St+1,Rt∣st,ati)=∑at−iπ−i(at−i∣st)P(St+1,Rt∣st,ati,at−i). As other agents' policies π−i change during learning, this local distribution can change even when the environment dynamics remain fixed. This gives rise to the non-stationarity (NS) experienced by each agent.
(2) Dynamics distribution modeling and context inference. DAC uses a VAE-like objective to learn dynamics-aware contexts from local transitions. Specifically, given a transition (st,ati,rt,st+1), the encoder infers the posterior distribution qenci(Zti∣st,ati,rt,st+1), from which a context zti is sampled. The decoder then takes (st,ati,zti) as input and predicts (st+1,rt).
At the generative-model level, marginalizing over all possible contexts defines a model of the full conditional dynamics distribution: Pθi(St+1,Rt∣st,ati)=∑zp0(z)pdec,θi(St+1,Rt∣st,ati,z), where p0(z) is the context prior and pdec,θi is the decoder's conditional distribution. Here, Pθi denotes the model distribution, and approximates the true local distribution Pπ−ii for fixed π−i.
The key distinction is that marginalization describes the model's distribution over all possible outcomes, whereas posterior inference produces a context associated with the particular outcome observed in the current transition. Under dynamics sufficiency and minimality (Assumptions 2 and 3 in Appendix B.1), for each fixed (st,ati), the inferred context zti and the observed outcome (st+1,rt) have a one-to-one correspondence.
(3) From the inferred context to a joint-action equivalence class. In a deterministic environment, for a fixed (st,ati), two joint actions at−i and a^t−i are dynamics-equivalent with respect to agent i if and only if they induce the same outcome (st+1,rt). The corresponding equivalence class is ct=[at−i]st,atii=a^t−i∣a^t−i∼st,atiiat−i.
Thus, given (st,ati), a specific outcome (st+1,rt) identifies exactly one equivalence class. Combining this observation with the context-outcome correspondence above gives:
zti↔(st+1,rt)↔ct,for each fixed (st,ati).
Therefore, although DAC never observes other agents' joint action, the context inferred from a specific local transition uniquely represents its dynamics-equivalence class under the stated assumptions. Accordingly, with a fixed context representation, conditioning on (st,ati,zti) determines the current reward rt and next state st+1, making the corresponding conditional outcome distribution stationary. Furthermore, for any fixed joint policy π, Theorem 2 establishes that there exists a context-based value function satisfying Qˉi,π(st,ati,zti)=Qπ(st,ati,at−i). These properties provide the rationale for conditioning value learning on the inferred context to address transition-level NS without observing other agents' actions.
(4) Value learning and optimistic action selection. DAC trains Qi(st,ati,zti) using contexts inferred from observed transitions. At action-selection time, the current outcome (st+1,rt) given (st,ati) is unavailable, so the corresponding context cannot yet be inferred. DAC therefore computes the optimistic marginal value and selects its greedy action as follows:
ϕi(st,ati)=maxzti∈ZiQi(st,ati,zti),ati∈argmaxaiϕi(st,ai).
This evaluates each local action under favorable joint actions of the other agents, aiming to alleviate the relative overgeneralization (RO) by avoiding value averaging over their sub-optimal or exploratory actions.
The finite discrete context space makes the maximization in (4) tractable; the equivalence-class interpretation in (3) depends on the stated theoretical assumptions and sufficient representation capacity. Below, we clarify the discrete design and its capacity limits, explain the connection to teammates' joint actions and the role of determinism, and discuss applicability under partial observability.