acceptodds
Under review as a conference paper at ICLR 2027

Fully Decentralized Cooperative Multi-Agent Reinforcement Learning is A Context Modeling Problem

Abstract

In this paper, we consider fully decentralized cooperative multi-agent reinforcement learning, wherein each agent solely observes the states, its local actions, and the shared team rewards. For each agent, the lack of access to other agents' actions typically induces non-stationarity during value function updates and relative overgeneralization during value function estimation, which together impede effective cooperative policy learning. However, existing works fail to address both issues simultaneously, due to their inability to model the joint action of other agents in a fully decentralized setting. To overcome this limitation, we propose Dynamics-Aware Context (DAC), which formalizes each agent's locally perceived task as a Contextual Markov Decision Process, and addresses both non-stationarity and relative overgeneralization through dynamics-aware context modeling. Specifically, DAC attributes variations in per-agent local task dynamics to switches among unobserved contexts, and instantiate them as latent variables that model the step-wise dynamics distribution. Theoretical analysis shows that each context uniquely represents an equivalence class of other agents' joint actions, thus allowing a context-based value function learned for each agent to address both non-stationarity and relative overgeneralization. Empirically, we evaluate DAC across various cooperative tasks, and its superior performance highlights its effectiveness.

Primary area:
reinforcement learning
Keywords:
Fully decentralized setting, Cooperative multi-agent reinforcement learning, Context modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hqdMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the non-stationarity and relative overgeneralization problems in cooperative multi-agent reinforcement learning. This paper proposes a fully decentralized framework that infers a discrete latent context from each agent’s sliding-window local trajectory, which aims to capture the teammate-induced environmental dynamics. Empirical validations are conducted on matrix games, SMAC, and a continuous differential game benchmark.

Critical strengths

This paper clearly identifies and formalizes the two core challenges, non-stationarity and relative overgeneralization, in cooperative MARL. It proposes to tackle both challenges by inferring a latent context that captures teammate-induced environment dynamics. The authors offer principled theoretical motivation by proving that the inferred latent context serves as a sufficient statistic of teammates’ joint actions under its assumptions.

Critical weaknesses
  1. The discrete context design relies on the claim that dynamics equivalence classes form a discrete set. This assumption is not justified for continuous teammate action spaces. The limited representational capacity of discrete context may restrict the method’s applicability.

  2. It is not clearly explained why context inferred from an agent’s recent local trajectory contains relevant information about teammates’ joint actions. In addition, Theorem 1 is built upon the assumption of a deterministic environment.

Questions for authors
  1. I remain confused about the definition and design of context. Why is context designed as discrete categories? Is this design motivated by modeling agent actions?

  2. Does the method still work if agents only have local partial observations and cannot access the global state?

5 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

We apologize for our delayed response, as an unexpected power outage in our server room disrupted our additional experiments.

We begin by revisiting the main workflow underlying our work in an accessible manner. For clarity, uppercase letters denote random variables, and lowercase letters denote their realizations.

(1) The generation of per-agent local dynamics. For each agent ii, its locally perceived transition and reward distribution is Pπ−ii(St+1,Rt∣st,ati)=∑at−iπ−i(at−i∣st)P(St+1,Rt∣st,ati,at−i)P_{\pi^{-i}}^i(S_{t+1},R_t|s_t,a_t^i)=\sum_{a_t^{-i}}\pi^{-i}(a_t^{-i}|s_t)P(S_{t+1},R_t|s_t,a_t^i,a_t^{-i}). As other agents' policies π−i\pi^{-i} change during learning, this local distribution can change even when the environment dynamics remain fixed. This gives rise to the non-stationarity (NS) experienced by each agent.

(2) Dynamics distribution modeling and context inference. DAC uses a VAE-like objective to learn dynamics-aware contexts from local transitions. Specifically, given a transition (st,ati,rt,st+1)(s_t,a_t^i,r_t,s_{t+1}), the encoder infers the posterior distribution qenci(Zti∣st,ati,rt,st+1)q_{\mathrm{enc}}^i(Z_t^i|s_t,a_t^i,r_t,s_{t+1}), from which a context ztiz_t^i is sampled. The decoder then takes (st,ati,zti)(s_t,a_t^i,z_t^i) as input and predicts (st+1,rt)(s_{t+1},r_t).

At the generative-model level, marginalizing over all possible contexts defines a model of the full conditional dynamics distribution: P^θi(St+1,Rt∣st,ati)=∑zp0(z)pdec,θi(St+1,Rt∣st,ati,z)\widehat P_\theta^i(S_{t+1},R_t|s_t,a_t^i)=\sum_z p_0(z)p_{\mathrm{dec},\theta}^i(S_{t+1},R_t|s_t,a_t^i,z), where p0(z)p_0(z) is the context prior and pdec,θip_{\mathrm{dec},\theta}^i is the decoder's conditional distribution. Here, P^θi\widehat P_\theta^i denotes the model distribution, and approximates the true local distribution Pπ−iiP_{\pi^{-i}}^i for fixed π−i\pi^{-i}.

The key distinction is that marginalization describes the model's distribution over all possible outcomes, whereas posterior inference produces a context associated with the particular outcome observed in the current transition. Under dynamics sufficiency and minimality (Assumptions 2 and 3 in Appendix B.1), for each fixed (st,ati)(s_t,a_t^i), the inferred context ztiz_t^i and the observed outcome (st+1,rt)(s_{t+1},r_t) have a one-to-one correspondence.

(3) From the inferred context to a joint-action equivalence class. In a deterministic environment, for a fixed (st,ati)(s_t,a_t^i), two joint actions at−ia_t^{-i} and a^t−i\hat a_t^{-i} are dynamics-equivalent with respect to agent ii if and only if they induce the same outcome (st+1,rt)(s_{t+1},r_t). The corresponding equivalence class is ct=[at−i]st,atii=a^t−i∣a^t−i∼st,atiiat−ic_t=[a_t^{-i}]_{s_t,a_t^i}^i=\\{\hat a_t^{-i}|\hat a_t^{-i}\sim_{s_t,a_t^i}^{i}a_t^{-i}\\}.

Thus, given (st,ati)(s_t,a_t^i), a specific outcome (st+1,rt)(s_{t+1},r_t) identifies exactly one equivalence class. Combining this observation with the context-outcome correspondence above gives:

zti↔(st+1,rt)↔ct,for each fixed (st,ati).z_t^i\leftrightarrow(s_{t+1},r_t)\leftrightarrow c_t, \quad \text{for each fixed }(s_t,a_t^i).

Therefore, although DAC never observes other agents' joint action, the context inferred from a specific local transition uniquely represents its dynamics-equivalence class under the stated assumptions. Accordingly, with a fixed context representation, conditioning on (st,ati,zti)(s_t,a_t^i,z_t^i) determines the current reward rtr_t and next state st+1s_{t+1}, making the corresponding conditional outcome distribution stationary. Furthermore, for any fixed joint policy π\boldsymbol{\pi}, Theorem 2 establishes that there exists a context-based value function satisfying Qˉi,π(st,ati,zti)=Qπ(st,ati,at−i)\bar Q^{i,\boldsymbol{\pi}}(s_t,a_t^i,z_t^i)=Q^{\boldsymbol{\pi}}(s_t,a_t^i,a_t^{-i}). These properties provide the rationale for conditioning value learning on the inferred context to address transition-level NS without observing other agents' actions.

(4) Value learning and optimistic action selection. DAC trains Qi(st,ati,zti)Q^i(s_t,a_t^i,z_t^i) using contexts inferred from observed transitions. At action-selection time, the current outcome (st+1,rt)(s_{t+1},r_t) given (st,ati)(s_t, a_t^i) is unavailable, so the corresponding context cannot yet be inferred. DAC therefore computes the optimistic marginal value and selects its greedy action as follows:

ϕi(st,ati)=max⁡zti∈ZiQi(st,ati,zti),ati∈arg⁡max⁡aiϕi(st,ai).\phi^i(s_t,a_t^i)=\max_{z_t^i\in\mathcal Z^i}Q^i(s_t,a_t^i,z_t^i), \qquad a_t^i\in\arg\max_{a^i}\phi^i(s_t,a^i).

This evaluates each local action under favorable joint actions of the other agents, aiming to alleviate the relative overgeneralization (RO) by avoiding value averaging over their sub-optimal or exploratory actions.

The finite discrete context space makes the maximization in (4) tractable; the equivalence-class interpretation in (3) depends on the stated theoretical assumptions and sufficient representation capacity. Below, we clarify the discrete design and its capacity limits, explain the connection to teammates' joint actions and the role of determinism, and discuss applicability under partial observability.

AuthorsOpenReview ↗

Rebuttal (Part 2)

For clarity, given (st,ati,rt,st+1)(s_t, a_t^i, r_t, s_{t+1}), we denote the context by ztiz_t^i and the equivalence class of the other agents' joint action by ctc_t. Here, ztiz_t^i denotes the same context as ctic_t^i in the main text; this notational change applies only to the rebuttal.

Weakness 1: The discrete context design relies on the claim that dynamics equivalence classes form a discrete set. This assumption is not justified for continuous teammate action spaces. The limited representational capacity of discrete context may restrict the method’s applicability.

1. Response to Weakness 1.

As described in point (4) of the workflow, DAC adopts a finite discrete context space to compute the optimistic marginal value ϕi(st,ati)=max⁡ztiQi(st,ati,zti)\phi^i(s_t,a_t^i)=\max_{z_t^i}Q^i(s_t,a_t^i,z_t^i) efficiently through enumeration. The discrete design serves this computational purpose; it does not imply that dynamics-equivalence classes are necessarily finite.

The exact correspondence in point (3) requires sufficient context capacity, together with the stated sufficiency and minimality assumptions. For a fixed (st,ati)(s_t,a_t^i), continuous teammate actions may induce infinitely many distinct outcomes and hence equivalence classes. In this case, a finite context vocabulary cannot preserve the exact correspondence and can only provide an approximate representation. We will clarify this capacity limitation and distinguish it from the computational motivation for choosing discrete contexts.

A possible extension to continuous contexts involves two components:

(1) Learn a bounded continuous context ztiz_t^i from local transition windows using a VAE-style objective that reconstructs rewards and next states.

(2) Learn a context-search network πci(st,ati)\pi_c^i(s_t,a_t^i) to approximately maximize Qi(st,ati,zti)Q^i(s_t,a_t^i,z_t^i), giving

\approx Q^i\\!\left(s_t,a_t^i,\pi_c^i(s_t,a_t^i)\right).$$ The search network could be optimized through gradients of the value function, following the deterministic actor-update principle used in Deep Deterministic Policy Gradient (DDPG). This would replace finite enumeration with learned context search. It would also require ensuring that proposed contexts are supported for the corresponding $(s_t,a_t^i)$, since bounding the latent space alone does not establish their validity. We leave the development and evaluation of this extension to future work. **Weakness 2:** It is not clearly explained why context inferred from an agent’s recent local trajectory contains relevant information about teammates’ joint actions. In addition, Theorem 1 is built upon the assumption of a deterministic environment. **2. Response to Weakness 2.** **2.1 Information about teammates’ joint actions in local trajectories.** As explained in points (2)–(3) of the workflow, teammates’ joint actions affect the reward and next state observed by agent $i$. Although $a_t^{-i}$ is unavailable, its effects appear in the local transition $(s_t,a_t^i,r_t,s_{t+1})$. The encoder infers $z_t^i$ from this transition, and reconstruction trains it to retain those effects. Under the stated assumptions, $$z_t^i\leftrightarrow(s_{t+1},r_t)\leftrightarrow c_t, \quad \text{for each fixed }(s_t,a_t^i).$$ Thus, information about teammates’ actions is obtained through their observable consequences. The context identifies their dynamics-equivalence class, which may contain multiple joint actions; recovering the exact action is unnecessary. **2.2 The role of deterministic dynamics.** Determinism supports the outcome-class correspondence in point (3). Given $(s_t,a_t^i,a_t^{-i})$, the reward and next state are fixed. In stochastic environments, equivalence classes can instead be defined by their full conditional outcome distributions. One class may then generate multiple outcomes, and different classes may generate the same outcome. When these distributions overlap, a specific local transition $(s_t,a_t^i,r_t,s_{t+1})$ can be consistent with multiple joint-action equivalence classes. Therefore, observing this transition does not, in general, uniquely determine its generating class $c_t$. Even if $z_t^i$ perfectly reconstructs the observed outcome and satisfies Assumptions 2 and 3, it cannot resolve this ambiguity: $$z_t^i\leftrightarrow(s_{t+1},r_t) \quad\not\Rightarrow\quad z_t^i\leftrightarrow c_t, \qquad \text{for each fixed }(s_t,a_t^i).$$ Thus, the one-to-one context-equivalence-class correspondence is not generally guaranteed in stochastic environments. The deterministic assumption in Theorem 1 allows a specific observed outcome to identify exactly one joint-action equivalence class, completing the correspondence established in the workflow.
AuthorsOpenReview ↗

Rebuttal (Part 3)

Question 1: I remain confused about the definition and design of context. Why is context designed as discrete categories? Is this design motivated by modeling agent actions?

3. Response to Question 1.

As explained in point (2) of the workflow, a context ztiz_t^i is a latent representation inferred from agent ii’s observed local transitions and trained through reward and next-state reconstruction. For k=1k=1, the encoder takes (st,ati,rt,st+1)(s_t,a_t^i,r_t,s_{t+1}), and the decoder reconstructs (st+1,rt)(s_{t+1},r_t) from (st,ati,zti)(s_t,a_t^i,z_t^i). Context categories are learned through this reconstruction objective, without observing or using labels for teammates’ actions.

Following point (4) of the workflow, we choose discrete categories to make optimistic action selection computationally tractable:

ϕi(st,ati)=max⁡zti∈ZiQi(st,ati,zti).\phi^i(s_t,a_t^i)=\max_{z_t^i\in\mathcal Z^i}Q^i(s_t,a_t^i,z_t^i).

A finite context space allows this maximum to be computed by enumeration. The discrete design is therefore motivated by this computation, rather than by assigning a category to each teammate joint action.

The connection to teammates’ actions follows from their effects on rewards and next states. Under the stated theoretical assumptions,

\qquad \text{for each fixed }(s_t,a_t^i).$$ Thus, a context identifies a dynamics-equivalence class that may contain several joint actions. Once the context is inferred, DAC can perform context-based value function learning without recovering the other agents’ exact joint actions. **Question 2:** Does the method still work if agents only have local partial observations and cannot access the global state? **4. Response to Question 2.** The current theoretical analysis does not cover general partially observable settings. As stated in points (2)-(3) of the workflow, the correspondence $$z_t^i\leftrightarrow(s_{t+1},r_t)\leftrightarrow c_t$$ is established for each fixed full state–action pair $(s_t,a_t^i)$. When only local observations are available, context inference instead uses $(o_t^i,a_t^i,r_t,o_{t+1}^i)$. The same $(o_t^i,a_t^i)$ may correspond to different hidden states. Furthermore, distinct global next states associated with different joint-action equivalence classes may produce the same local observation and reward. Consequently, a specific observed transition cannot generally identify the underlying class $c_t$. Even if the context–observed-outcome correspondence holds, $$z_t^i\leftrightarrow(o_{t+1}^i,r_t) \quad\not\Rightarrow\quad z_t^i\leftrightarrow c_t.$$ Partial observability also affects the value-consistency argument: identical observed outcomes may conceal different global next states with different values. Thus, reconstructing $(o_{t+1}^i,r_t)$ does not ensure that the context contains the information required to represent the corresponding joint action value.
Reviewer_aK6oOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Considering fully decentralized cooperative multi-agent reinforcement learning, where each agent observes the states, its local actions, and the shared team rewards this manuscript proposes Dynamics-Aware Context (DAC), which formalizes each agent’s locally perceived task as a Contextual Markov Decision Process. The proposal aims to address non-stationarity and relative over-generalization through dynamics-aware context modelling. DAC instantiates variations in unobserved agent contexts as latent variables that model the dynamics distribution. In so doing, each agent learns a context-based value function. Authors provide a theoretical analysis showing that each context uniquely represents an equivalence class of other agents’ joint actions. DAC is evaluated across various cooperative tasks, showing its merits.

Critical strengths
  • The paper formulates the problem addressed concisely and provides a theoretical analysis showing that each context uniquely represents an equivalence class of other agents’ joint actions.
  • Each agent is trained to infer compact representations of task dynamics distributions, using latent variables, by means of a VAE architecture
  • Authors clearly specify the limitations of their approach.
Critical weaknesses
  • While authors initially specify a fully observable setting in which agents share the same team reward, they formulate Contextual MDP so as to induce individual reward dynamics for each agent. The same happens in the theoretical analysis defining rtr_t, which is used subsequently in formulating the objective of context-based value learning. This needs to be clarified.
  • Choosing the optimistic marginal value needs further justification. It is not clear that this is the best choice in any case. It seems rather as an assumption or a heuristic. E.g. in cases of non-fully-cooperative settings this may not be the best choice. Authors also claim that this may lead to unreliable marginal values, given out-of-distribution contexts. These issues need further discussion and clarification.
  • Comparison to other works aiming at fully-collaborative MARL settings, where agents infer task dynamics using VAE-style architectures is missing (e.g. [1], [2], [3]): These proposals constrain the innovation of the submission.
  • Finally, the scalability of the method in terms of sample-efficiency and number of agents involved is not addressed.

Minor: The same pip^i is used to denote different distributions.


[1] Andreas Kontogiannis, Konstantinos Papathanasiou, Yi Shen, Giorgos Stamou, Michael M. Zavlanos, and George Vouros. 2025. Enhancing cooperative multi-agent reinforcement learning with state modelling and adversarial exploration. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267. JMLR.org, Article 1235, 31437–31466.

[2] Papoudakis, G., Christianos, F., and Albrecht, S. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210–19222, 2021.

[3] Xu, Z., Bai, Y., Li, D., Zhang, B., and Fan, G. Side: State inference for partially observable cooperative multi-agent reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 1400–1408, 2022.

Questions for authors

Can you please address each of the weaknesses mentioned above?

8 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

We apologize for our delayed response, as an unexpected power outage in our server room disrupted our additional experiments.

Weakness 1: While authors initially specify a fully observable setting in which agents share the same team reward, they formulate Contextual MDP so as to induce individual reward dynamics for each agent. The same happens in the theoretical analysis defining rtr_t, which is used subsequently in formulating the objective of context-based value learning. This needs to be clarified.

1. Response to Weakness 1.

All agents receive the same realized team reward rtr_t. Under the deterministic setting, a joint transition (st,at,rt,st+1)(s_t,\boldsymbol a_t,r_t,s_{t+1}) satisfies rt=R(st,at)r_t=R(s_t,\boldsymbol a_t); DAC does not introduce separate individual rewards. Here, at=ati,at−i\boldsymbol{a}_{t}=\\{a_t^i,a_t^{-i}\\} denotes all agents' joint action and RR is the reward function of the multi-agent environment.

Full observability means that each agent observes sts_t, but fully decentralized learning does not provide access to other agents' actions at−ia_t^{-i}. Consequently, Eq. (1) in the main text (line 140-143) defines the reward function perceived by agent ii by marginalizing these unobserved actions:

Ri(st,ati)=∑at−iπ−i(at−i∣st)R(st,at).R^i(s_t,a_t^i)=\sum_{a_t^{-i}}\pi^{-i}(a_t^{-i}|s_t)R(s_t,\boldsymbol a_t).

As π−i\pi^{-i} changes, the locally perceived reward dynamics can change even though the shared environment reward function RR remains fixed. This is the meaning of agent-conditioned reward dynamics in the local MDP / CMDP formulation. The theoretical analysis and the TD objective in context-based value learning consistently use the observed shared reward rtr_t. We will clarify this distinction in the revision.

AuthorsOpenReview ↗

Rebuttal (Part 2)

Weakness 2: Choosing the optimistic marginal value needs further justification. It is not clear that this is the best choice in any case. It seems rather as an assumption or a heuristic. E.g. in cases of non-fully-cooperative settings this may not be the best choice. Authors also claim that this may lead to unreliable marginal values, given out-of-distribution contexts. These issues need further discussion and clarification.

2. Response to Weakness 2.

2.1 DAC’s workflow and the justification for optimistic action selection.

Note: For clarity, given (st,ati,rt,st+1)(s_t, a_t^i, r_t, s_{t+1}), we denote the context by ztiz_t^i and the equivalence class of the other agents' joint action by ctc_t. Here, ztiz_t^i denotes the same context as ctic_t^i in the main text; this notational change applies only to the rebuttal.

This work focuses on fully decentralized cooperative multi-agent reinforcement learning, where all agents share the same team reward. During learning, DAC infers ztiz_t^i from recent local transitions containing (st,ati,rt,st+1)(s_t,a_t^i,r_t,s_{t+1}), uses the context to reconstruct transition outcomes, and trains the context-based value function Qi(st,ati,zti)Q^i(s_t,a_t^i,z_t^i). At action-selection time, the current outcome (rt,st+1)(r_t,s_{t+1}) is unavailable, so its context cannot yet be inferred. DAC therefore evaluates candidate actions through

ϕi(st,ati)=max⁡zti∈ZiQi(st,ati,zti),ati∈arg⁡max⁡aiϕi(st,ai).\phi^i(s_t,a_t^i)=\max_{z_t^i\in\mathcal Z^i}Q^i(s_t,a_t^i,z_t^i), \quad a_t^i\in\arg\max_{a^i}\phi^i(s_t,a^i).

The justification follows from the context–equivalence-class correspondence. In the single-transition analysis, deterministic, fully observable dynamics and sufficient, minimal context representations imply that each context identifies an equivalence class ctc_t of other agents’ joint actions producing the same (rt,st+1)(r_t,s_{t+1}) for a fixed (st,ati)(s_t,a_t^i). For realizable contexts Zi(st,ati)\mathcal Z^i(s_t,a_t^i), the ideal value satisfies

max⁡zti∈Zi(st,ati)Qˉi,π(st,ati,zti)=max⁡at−iQπ(st,ati,at−i).\max_{z_t^i\in\mathcal Z^i(s_t,a_t^i)}\bar Q^{i,\pi}(s_t,a_t^i,z_t^i)=\max_{a_t^{-i}}Q^\pi(s_t,a_t^i,a_t^{-i}).

Thus, context maximization evaluates an action under favorable feasible coordination. A conventional local action value averages over teammates’ actions; RO occurs when teammates' sub-optimal actions lower this average enough to make a potentially superior cooperative action appear inferior. Context maximization prevents those unfavorable classes from directly lowering its score through averaging.

This provides a mechanism for mitigating RO in fully cooperative tasks. In non-fully-cooperative settings, other agents may pursue conflicting objectives, so assuming their most favorable behavior is generally unjustified; our claim is restricted to fully cooperative tasks.

2.2 Out-of-distribution (OOD) contexts and possible improvements.

Context support depends on (st,ati)(s_t,a_t^i): a context observed elsewhere may be unsupported for the current state–action pair. This affects two stages:

(1) Optimistic action selection. In ϕi(st,ati)=max⁡ztiQi(st,ati,zti)\phi^i(s_t,a_t^i)=\max_{z_t^i}Q^i(s_t,a_t^i,z_t^i), maximization can favor overestimated values associated with poorly supported (st,ati,zti)(s_t,a_t^i,z_t^i) combinations.

(2) Value function updates. Although zt+1iz_{t+1}^i is inferred from the actual next transition, maximizing over at+1ia_{t+1}^i can pair this context with alternative actions for which its value is poorly supported. Maximizing over both actions and contexts expands the combinations evaluated. Estimation errors in these combinations can propagate through bootstrapping, as discussed in our Response to Question 2 of Reviewer 4WSt.

Possible improvements include estimating context support conditioned on (st,ati)(s_t,a_t^i), restricting maximization to supported combinations, and incorporating uncertainty penalties into value estimates. Such restrictions should preserve rare but valuable cooperative outcomes. We are exploring these directions in our ongoing work to improve the reliability of both optimistic action selection and value updates.

AuthorsOpenReview ↗

Rebuttal (Part 3)

Weakness 3: Comparison to other works aiming at fully-collaborative MARL settings, where agents infer task dynamics using VAE-style architectures is missing (e.g. [1], [2], [3]): These proposals constrain the innovation of the submission.

3. Response to Weakness 3.

3.1 Distinguishing research objectives and the role of learned representations.

There may be a potential misunderstanding regarding this work. In general, the relationship between research works should be assessed through the problems addressed and use of the inferred representations. Architectural similarities such as VAE-style modelling alone do not establish equivalent contributions. Specifically, we summarize these three works and DAC below:

  • SMPE² [1] addresses partial observability and exploration by learning task-relevant state beliefs from local observations and using them to improve policies and guide adversarial exploration.

  • LIAM [2] addresses agent modelling under partial observability. It reconstructs other agents' observations and actions during training, enabling a controlled agent to infer their behaviour from its own local history during execution. Its original setting considers other agents drawn from a fixed policy set.

  • SIDE [3] addresses unavailable global states during centralized training. It combines all agents' local histories to infer a latent state for value decomposition, instantiated with QMIX.

Different from them, DAC focuses on transition-level NS and RO in fully decentralized cooperative MARL. It infers contexts from local transitions (st,ati,rt,st+1)(s_t,a_t^i,r_t,s_{t+1}), conditions per-agent decentralized value function learning through Qi(st,ati,zti)Q^i(s_t,a_t^i,z_t^i), and uses max⁡ztiQi(st,ati,zti)\max_{z_t^i}Q^i(s_t,a_t^i,z_t^i) for optimistic action selection. Under the stated assumptions, these contexts identify joint-action equivalence classes.

DAC's contribution lies in the problem it addresses, transition-level NS and RO in fully decentralized cooperative MARL, and the connection it establishes among transition reconstruction, context–joint-action equivalence, context-based value function learning, and optimistic action selection; the VAE-style architecture serves only as a technical means of realizing this approach.

3.2 Supplementary comparisons on tasks with severe RO.

We are comparing DAC with adapted implementations of SMPE² and SIDE on the matrix game and predator-prey tasks with P=0.3,0.5,1.0P=0.3,0.5,1.0. Their EPyMARL/PyMARL implementations facilitate integration into our PyMARL codebase. As shown in https://postimg.cc/K3YNdNQN, SMPE² and SIDE fail to efficiently learn optimal policies in these tasks, suggesting that improving state inference and exploration does not automatically resolve RO. As discussed in Response 3.1, their primary objectives and training requirements differ from DAC’s fully decentralized treatment of NS and RO. These experiments therefore provide supplementary comparisons; treating methods addressing different research problems as directly matched baselines would be inappropriate.

LIAM uses a separate A2C-based implementation and a different agent-modelling training setup, requiring additional adaptation and validation for our PyMARL benchmark. The short rebuttal period and tight deadline leave insufficient time to complete this process reliably. We therefore include LIAM in the methodological discussion and defer its experimental comparison.

Reference:

[1] Andreas Kontogiannis, Konstantinos Papathanasiou, Yi Shen, Giorgos Stamou, Michael M. Zavlanos, and George Vouros. 2025. Enhancing cooperative multi-agent reinforcement learning with state modelling and adversarial exploration. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267. JMLR.org, Article 1235, 31437–31466.

[2] Papoudakis, G., Christianos, F., and Albrecht, S. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210–19222, 2021.

[3] Xu, Z., Bai, Y., Li, D., Zhang, B., and Fan, G. Side: State inference for partially observable cooperative multi-agent reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 1400–1408, 2022.

Reviewer_4WStOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies fully decentralized cooperative MARL. Each agent sees only the state, its own action, and the team reward. The method (DAC) encodes a sliding window of recent local transitions into a discrete latent context. Each agent learns a Q-function that is conditioned on this context. Each agent selects actions by a maximum over contexts. The authors claim that this addresses non-stationarity (NS) and relative overgeneralization (RO) together. Experiments include a matrix game, predator-prey, SMAC, SMACv2, and a differential game.

Critical strengths
  • The key idea is simple and good. A learned latent context gives each agent a decentralized way to represent the effect of the other agents. The maximum over contexts then gives an optimistic value.
  • The method seems to help where RO is severe. The clearest results are the matrix game and predator-prey at P = 1.0, where only DAC learns a good policy. Table 1 shows that the learned contexts recover the payoff structure.
  • The paper states its limitations clearly, for example the loss of advantage on SMACv2.
Critical weaknesses
  1. The relation to prior work is not clear. Under Assumptions 2 and 3, the context is a label for the realized outcome (r, s') at a given (s, a^i). Then max_c Q(s, a^i, c) is approximately a maximum of r + γV(s') over the reachable outcomes. This is the same idea as distributed Q-learning, I2Q, Best Possible Q-learning (Jiang & Lu, UAI 2025), and MaxMax Q-learning (Zhu et al., 2024). The paper does not cite BQL or MMQ. The paper does not compare with distributed Q-learning, BQL, or MMQ. Also, I2Q and BQL already claim to address NS and RO together, so the claim of a first unified treatment is not correct. This is the main reason for my uncertainty. I cannot tell what the learned context adds beyond these methods.
  2. The theory adds little. Assumptions 2 and 3 make c a bijective label of the outcome, so Theorems 1 and 2 follow from the definitions. The theory also assumes deterministic, fully observable dynamics, so it does not cover SMAC or k > 1.
  3. The context has two meanings. Section 4.1 says that the policy of the other agents induces the context. Section 4.3 says that the context represents the per-step joint action. The sliding-window and concept-drift motivation supports only the first meaning.
  4. The experiments do not isolate the contribution. No ablation replaces the learned context with an exact outcome hash, which the Limitations section itself suggests. No experiment isolates the NS claim. The paper does not state the number of seeds or what the shaded regions show.
Questions for authors
  1. What does DAC do that a maximum over a learned discretization of (r, s') does not do? A comparison with distributed Q-learning or MMQ, or an outcome-hash ablation, would answer this.
  2. In Eq. 7, the target takes a maximum over a^i_{t+1} with c_{t+1} fixed, but the encoder inferred c_{t+1} from the action that the agent actually took. Why is this target correct?
  3. How many seeds did you use, and what do the shaded regions show?
10 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

We apologize for our delayed response, as an unexpected power outage in our server room disrupted our additional experiments.

Summary of DAC's workflow

We begin by revisiting the main workflow underlying our work in an accessible manner. For clarity, uppercase letters denote random variables, and lowercase letters denote their realizations.

(1) The generation of per-agent local dynamics. For each agent ii, its locally perceived transition and reward distribution is Pπ−ii(St+1,Rt∣st,ati)=∑at−iπ−i(at−i∣st)P(St+1,Rt∣st,ati,at−i)P_{\pi^{-i}}^i(S_{t+1},R_t|s_t,a_t^i)=\sum_{a_t^{-i}}\pi^{-i}(a_t^{-i}|s_t)P(S_{t+1},R_t|s_t,a_t^i,a_t^{-i}). As other agents' policies π−i\pi^{-i} change during learning, this local distribution can change even when the environment dynamics remain fixed. This gives rise to the non-stationarity (NS) experienced by each agent.

(2) Dynamics distribution modeling and context inference. DAC uses a VAE-like objective to learn dynamics-aware contexts from local transitions. Specifically, given a transition (st,ati,rt,st+1)(s_t,a_t^i,r_t,s_{t+1}), the encoder infers the posterior distribution qenci(Zti∣st,ati,rt,st+1)q_{\mathrm{enc}}^i(Z_t^i|s_t,a_t^i,r_t,s_{t+1}), from which a context ztiz_t^i is sampled. The decoder then takes (st,ati,zti)(s_t,a_t^i,z_t^i) as input and predicts (st+1,rt)(s_{t+1},r_t).

At the generative-model level, marginalizing over all possible contexts defines a model of the full conditional dynamics distribution: P^θi(St+1,Rt∣st,ati)=∑zp0(z)pdec,θi(St+1,Rt∣st,ati,z)\widehat P_\theta^i(S_{t+1},R_t|s_t,a_t^i)=\sum_z p_0(z)p_{\mathrm{dec},\theta}^i(S_{t+1},R_t|s_t,a_t^i,z), where p0(z)p_0(z) is the context prior and pdec,θip_{\mathrm{dec},\theta}^i is the decoder's conditional distribution. Here, P^θi\widehat P_\theta^i denotes the model distribution, and approximates the true local distribution Pπ−iiP_{\pi^{-i}}^i for fixed π−i\pi^{-i}.

The key distinction is that marginalization describes the model's distribution over all possible outcomes, whereas posterior inference produces a context associated with the particular outcome observed in the current transition. Under dynamics sufficiency and minimality (Assumptions 2 and 3 in Appendix B.1), for each fixed (st,ati)(s_t,a_t^i), the inferred context ztiz_t^i and the observed outcome (st+1,rt)(s_{t+1},r_t) have a one-to-one correspondence.

(3) From the inferred context to a joint-action equivalence class. In a deterministic environment, for a fixed (st,ati)(s_t,a_t^i), two joint actions at−ia_t^{-i} and a^t−i\hat a_t^{-i} are dynamics-equivalent with respect to agent ii if and only if they induce the same outcome (st+1,rt)(s_{t+1},r_t). The corresponding equivalence class is ct=[at−i]st,atii=a^t−i∣a^t−i∼st,atiiat−ic_t=[a_t^{-i}]_{s_t,a_t^i}^i=\\{\hat a_t^{-i}|\hat a_t^{-i}\sim_{s_t,a_t^i}^{i}a_t^{-i}\\}.

Thus, given (st,ati)(s_t,a_t^i), a specific outcome (st+1,rt)(s_{t+1},r_t) identifies exactly one equivalence class. Combining this observation with the context-outcome correspondence above gives:

zti↔(st+1,rt)↔ct,for each fixed (st,ati)z_t^i\leftrightarrow(s_{t+1},r_t)\leftrightarrow c_t, \text{for each fixed }(s_t,a_t^i).

Therefore, although DAC never observes other agents' joint action, the context inferred from a specific local transition uniquely represents its dynamics-equivalence class under the stated assumptions. Accordingly, with a fixed context representation, conditioning on (st,ati,zti)(s_t,a_t^i,z_t^i) determines the current reward rtr_t and next state st+1s_{t+1}, making the corresponding conditional outcome distribution stationary. Furthermore, for any fixed joint policy π\boldsymbol{\pi}, Theorem 2 establishes that there exists a context-based value function satisfying Qˉi,π(st,ati,zti)=Qπ(st,ati,at−i)\bar Q^{i,\boldsymbol{\pi}}(s_t,a_t^i,z_t^i)=Q^{\boldsymbol{\pi}}(s_t,a_t^i,a_t^{-i}). These properties provide the rationale for conditioning value learning on the inferred context to address transition-level NS without observing other agents' actions.

(4) Value learning and optimistic action selection. DAC trains Qi(st,ati,zti)Q^i(s_t,a_t^i,z_t^i) using contexts inferred from observed transitions. At action-selection time, the current outcome (st+1,rt)(s_{t+1},r_t) given (st,ati)(s_t, a_t^i) is unavailable, so the corresponding context cannot yet be inferred. DAC therefore computes the optimistic marginal value and selects its greedy action as follows:

ϕi(st,ati)=max⁡zti∈ZiQi(st,ati,zti),ati∈arg⁡max⁡aiϕi(st,ai).\phi^i(s_t,a_t^i)=\max_{z_t^i\in\mathcal Z^i}Q^i(s_t,a_t^i,z_t^i), \qquad a_t^i\in\arg\max_{a^i}\phi^i(s_t,a^i).

This evaluates each local action under favorable joint actions of the other agents, aiming to alleviate the relative overgeneralization (RO) by avoiding value averaging over their sub-optimal or exploratory actions.

Below, we respond to each concern raised by the reviewer. For clarity, given (st,ati,rt,st+1)(s_t, a_t^i, r_t, s_{t+1}), we denote the context by ztiz_t^i and the equivalence class of the other agents' joint action by ctc_t. Here, ztiz_t^i denotes the same context as ctic_t^i in the main text; this notational change applies only to the rebuttal.

AuthorsOpenReview ↗

Rebuttal (Part 2)

Weakness 1: The relation to prior work is not clear. Under Assumptions 2 and 3, the context is a label for the realized outcome (r, s') at a given (s, aia^i). Then max⁡cQ(s,ai,c)\max_c Q(s, a^i, c) is approximately a maximum of r+γV(s′)r + \gamma V(s') over the reachable outcomes. This is the same idea as distributed Q-learning, I2Q, Best Possible Q-learning (Jiang & Lu, UAI 2025), and MaxMax Q-learning (Zhu et al., 2024). The paper does not cite BQL or MMQ. The paper does not compare with distributed Q-learning, BQL, or MMQ. Also, I2Q and BQL already claim to address NS and RO together, so the claim of a first unified treatment is not correct. This is the main reason for my uncertainty. I cannot tell what the learned context adds beyond these methods.

1. Response to Weakness 1.

1.1 The difference to prior works

Under the stated assumptions, the reviewer's outcome-label interpretation is correct: for each fixed (st,ati)(s_t,a_t^i), the inferred context satisfies zti↔(st+1,rt)z_t^i\leftrightarrow(s_{t+1},r_t). Let Zi(st,ati)\mathcal Z^i(s_t,a_t^i) and Oi(st,ati)\mathcal O^i(s_t,a_t^i) denote the realizable contexts and reachable outcomes, respectively. If Qi(st,ati,zti)Q^i(s_t,a_t^i,z_t^i) approximates the ideal context-based value associated with a fixed value function VV, then

Q^i(s_t,a_t^i,z_t^i) \\;\approx\\; \max_{(s_{t+1},r_t)\in\mathcal O^i(s_t,a_t^i)} \left[r_t+\gamma V(s_{t+1})\right].$$ The relation is exact for the corresponding ideal values. DAC therefore shares the optimistic interpretation of evaluating a local action under its most favorable reachable outcome. The distinction lies in how teammates’ effects on rewards and next states are represented and used in value learning. **DAC learns and retains context-based values from observed transitions, while context maximization is applied during action selection.** These methods can be compared through the same pipeline: constructing a representation from local transitions, updating the action-value function, and introducing optimism over other agents' effects. For compactness, let $V^i(s_{t+1}) = \max_{a_{t+1}^i}Q^i(s_{t+1},a_{t+1}^i), \qquad G_t^i=r_t+\gamma V^i(s_{t+1}).$ Using this notation, the table in **Rebuttal (Part 3)** compares how these methods learn representations, update value functions, and introduce optimism. These methods share optimism about teammates' actions but differ in how their effects are represented and where optimism is applied. Distributed Q-learning captures these effects implicitly through sampled TD targets and retains upward value updates. I2Q models an ideal next state guided by its state-state critic and constructs learning targets from that prediction. BQL estimates expected TD targets under the changing replay distribution and uses asymmetric updates to favor higher estimates. MMQ learns next-state bounds and maximizes the TD target over candidate states sampled within those bounds. Thus, these four methods incorporate optimistic assessments of teammates’ effects into learning $Q^i(s_t,a_t^i)$. DAC differs from them in two aspects: (1) DAC infers $z_t^i$ from $(s_t,a_t^i,r_t,s_{t+1})$ and reconstructs $(s_{t+1},r_t)$ from $(s_t,a_t^i,z_t^i)$, thereby explicitly encoding the effects of teammates' sampled joint action on the current reward and next state. Under the stated assumptions, $z_t^i$ identifies the corresponding equivalence class $c_t$ for each fixed $(s_t,a_t^i)$, making the conditional current-outcome distribution stationary. (2) DAC learns these effects' values separately through $Q^i(s_t,a_t^i,z_t^i)$. Context maximization enters action selection through $\phi^i(s_t,a_t^i)=\max_{z_t^i}Q^i(s_t,a_t^i,z_t^i)$, favoring local actions under their most favorable teammate-action classes to alleviate RO. We will add this analysis in the revision to further enhance the novelty of DAC. **1.2 Comparison to Distributed Q-learning and MaxMax Q-learning** The original experiments already include I2Q. We add comparisons with Distributed Q-learning (Distributed-Q) and MaxMax Q-learning (MaxMax-Q) within our framework. The two added baselines use 3 random seeds each due to the limited rebuttal period; DAC and Hysteretic-Q use 5 seeds each. We implement a neural adaptation of Distributed-Q by setting the negative-TD-error weight in our Hysteretic-Q implementation to zero, and adapt MaxMax-Q from the authors' public implementation for MPE and Differential Games. We have not located a public BQL implementation, and thus the comparison with BQL above is methodological. As shown in [https://postimg.cc/HjBMBLk3](https://postimg.cc/HjBMBLk3), Distributed-Q reaches the optimal return in the matrix game but performs poorly on other tasks. MaxMax-Q underperforms DAC across the evaluated tasks, with near-zero win rates on all four SMAC scenarios. Please refer to **Rebuttal (Part 3)** for the remainder of our response.
AuthorsOpenReview ↗

Rebuttal (Part 3)

MethodRepresentation learned from local transitionsValue update and location of optimism
Distributed Q-learningEach observed transition provides a sampled target GtiG_t^i. No dynamics model is learned.The tabular update is Qi(st,ati)←max⁡Qi(st,ati),Gti.\begin{aligned} Q^i(s_t,a_t^i)\leftarrow\max\\{Q^i(s_t,a_t^i), G_t^i\\}. \end{aligned} Retains the largest sampled backup encountered for each local state-action pair.
I2QLearns a state-state critic QssiQ_{\mathrm{ss}}^i by TD learning and an ideal next-state predictor s^t+1=fi(st,ati)\hat s_{t+1}=f^i(s_t,a_t^i). The predictor maximizes E![λQssi(st,s^t+1)−∥s^t+1−st+1∥22].\mathbb E\\!\left[ \begin{aligned} \lambda Q_{\mathrm{ss}}^i(s_t,\hat s_{t+1})-\lVert\hat s_{t+1}-s_{t+1}\rVert_2^2 \end{aligned} \right]. The first value term favors valuable next states; the second reconstruction term keeps predictions close to observed next states.Fits Qi(st,ati)Q^i(s_t,a_t^i) using targets evaluated at the predicted ideal next state s^t+1\hat s_{t+1}: For rt=R(st)r_t=R(s_t): rt+γmax⁡at+1iQˉi(s^t+1,at+1i)r_t+\gamma\max_{a_{t+1}^i}\bar Q^i(\hat s_{t+1},a_{t+1}^i). For rt=R(st,st+1)r_t=R(s_t,s_{t+1}): Qˉssi(st,s^t+1)\bar Q_{\mathrm{ss}}^i(s_t,\hat s_{t+1}), which includes the predicted transition's reward and next state value. Optimism enters value learning through the ideal-transition targets.
BQLAs teammate policies change, the replay distribution evolves. An auxiliary critic estimates the conditional mean of sampled TD targets under the current replay distribution: Qei(st,ati)≈EDi[Gti∣st,ati]Q_{\mathrm e}^i(s_t,a_t^i)\approx\mathbb E_{\mathcal D^i}[G_t^i\mid s_t,a_t^i]. It does not explicitly model individual transition kernels.The ideal operator computes max⁡PiEPi[Gti]\max_{P^i}\mathbb E_{P^i}[G_t^i] over feasible local transition kernels induced by teammates' policies. In practice, weighted regression moves QiQ^i toward the target-network estimate Qˉei\bar Q_{\mathrm e}^i, with weight 11 for upward targets and a smaller positive weight for downward targets. This favors higher expected targets encountered as replay changes and serves as a practical approximation to the ideal maximization.
MMQTrains next-state quantile networks g.05i(st,ati),g.95i(st,ati).g_{.05}^i(s_t,a_t^i), g_{.95}^i(s_t,a_t^i). Samples candidates uniformly within their predicted coordinatewise bounds and adds the observed st+1s_{t+1}, forming S^i(st,ati)\widehat{\mathcal S}^i(s_t,a_t^i). A reward model minimizes E![(R^i(st,st+1)−rt)2].\mathbb E\\!\left[\bigl(\widehat R^i(s_t,s_{t+1})-r_t\bigr)^2\right].Fits Qi(st,ati)Q^i(s_t,a_t^i) to max⁡s~t+1∈S^i(st,ati)[R^i(st,s~t+1)+γVi(s~t+1)].\max_{\substack{ \tilde s_{t+1}\in\widehat{\mathcal S}^i(s_t,a_t^i) }} \left[ \begin{aligned} \widehat R^i(s_t,\tilde s_{t+1})+\gamma V^i(\tilde s_{t+1}) \end{aligned} \right]. Optimism enters through selection of the highest-valued candidate next state in the TD target.
DACInfers a transition-specific context: zti∼qenci(⋅∣st,ati,rt,st+1)z_t^i\sim q_{\mathrm{enc}}^i(\cdot\mid s_t,a_t^i,r_t,s_{t+1}). Learns contexts through reward and next-state reconstruction. Under the stated assumptions, zti↔ctz_t^i\leftrightarrow c_t for each fixed (st,ati)(s_t,a_t^i).Fits the context-conditioned value Qi(st,ati,zti)Q^i(s_t,a_t^i,z_t^i) to the TD target: rt+γmax⁡at+1iQi(st+1,at+1i,zt+1i)\begin{aligned} r_t+\gamma\max_{a_{t+1}^i}Q^i(s_{t+1},a_{t+1}^i,z_{t+1}^i) \end{aligned}. Greedy action selection maximizes ϕi(st,ati)=max⁡ztiQi(st,ati,zti).\begin{aligned} \phi^i(s_t,a_t^i)=\max_{z_t^i}Q^i(s_t,a_t^i,z_t^i). \end{aligned} Context conditioning addresses transition-level NS; context maximization provides optimism for action selection.

The following is a continuation of Response 1.2 in Rebuttal (Part 2).

For Distributed-Q, function approximation and bootstrapping can introduce overestimated values, producing negative TD errors that require downward correction. Discarding these errors removes that direct correction signal. Allowing downward updates with a smaller weight, as in Hysteretic-Q, substantially improves performance in these experiments, although it does not consistently match DAC.

For MaxMax-Q, sampling within coordinatewise quantile bounds does not enforce joint transition feasibility. Consequently, candidate next states can be unreachable or outside the training-data distribution. Maximizing the TD target over these candidates can amplify reward and value extrapolation errors, which may contribute to the observed poor performance.

We will add the missing BQL and MaxMax-Q references and remove the claim of the first unified treatment of NS and RO. DAC’s contribution lies in learning reconstruction-based dynamics-aware contexts that, under the stated assumptions, uniquely represent other agents’ joint-action equivalence classes for each fixed (st,ati)(s_t,a_t^i). This correspondence supports context-conditioned value learning to address transition-level NS and context maximization for optimistic action selection to alleviate RO.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A brief summary of key updates

Dear AC and Reviewers,

For clarity, we summarize the key updates made during the rebuttal:

(1) Additional comparisons with Distributed Q-learning and MaxMax Q-learning, together with a detailed methodological comparison against existing optimistic learning methods. (Reviewer 4WSt)

(2) Additional DAC-Hash and fixed observation-encoding ablations, together with DAC (k=1k=1), to examine learned contexts and reconstruction across recent transitions. (Reviewer 4WSt)

(3) A controlled non-stationarity (NS) experiment with 10 matched seeds. Under teammate-policy switching, the final mean conditional-distribution shift is close to zero, supporting substantial alleviation of transition-level NS in this task. (Reviewer 4WSt)

(4) Additional DAC-Max comparisons evaluating maximization over both actions and contexts in the bootstrap target, supporting the practical choice of retaining the inferred context. (Reviewers 4WSt, aK6o)

(5) Additional comparisons with SMPE² and SIDE, alongside clarification of their differences from DAC in research problems and method details. (Reviewer aK6o)

(6) Additional training-time measurements and analysis of computational overhead and scalability. (Reviewer aK6o)

With respect to the reviewers' main concerns and follow-up questions:

(a) For Reviewer 4WSt:

(i) We clarify the theoretical contribution: under the stated assumptions, Theorem 1 establishes the context–joint-action equivalence correspondence, while Theorem 2 establishes preservation of joint-action values under a fixed joint policy. These results provide the rationale for DAC's context-based framework. (Response to Weakness 2)

(ii) We clarify DAC's differences from existing optimistic learning methods, particularly its retention of separate context-based values and use of context maximization during action selection, supported by the additional comparisons in update (1). (Responses to Weakness 1 and Question 1)

(iii) We provide the DAC-Hash, fixed observation-encoding, and DAC (k=1k=1) ablations, together with the controlled NS experiment in updates (2)–(3), to examine the contribution of context modeling and validate its effectiveness in alleviating transition-level NS. (Responses to Weaknesses 2 and 4 and Question 1)

(b) For Reviewer aK6o:

(i) We provide a definition table and numerical example to clarify the relationship between the shared team reward, the environment reward function, and each agent's local expected reward function. (Response to Reply 1)

(ii) We distinguish cooperative optimism from value overestimation and explain how unsupported action–context combinations affect action selection and value updates. The DAC-Max comparison in update (4) supports retaining the inferred context as a practical choice for the bootstrap target. (Response to Reply 2)

(iii) We provide a Bellman-based derivation and matrix-game example showing why improved state inference and exploration do not by themselves resolve RO. We explain how DAC addresses the averaging mechanism through separate context-based values and optimistic action selection. (Response to Reply 3)

We hope this summary can facilitate the discussion process and reduce your workload.

Thank you very much for the time and effort you have devoted to leading the discussions.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.