acceptodds
Under review as a conference paper at ICLR 2027

Disentangle to Correct: Separating Temporal Structure from Noise in Multi-Agent Reinforcement Learning

Abstract

Although the centralized training with decentralized execution (CTDE) paradigm has achieved notable success in multi-agent reinforcement learning, existing methods still struggle with robustness and long-horizon credit assignment, largely due to the entangled modeling of stable temporal structure and transient noise. To address this issue, we propose Structure–Noise Disentanglement (SND), a framework that explicitly factorizes agent representations into a structural branch for modeling long-term coordination structure and a noise branch for capturing instantaneous residuals. The structural branch employs Mamba, a selective state-space model, to efficiently capture reward-relevant long-range dependencies, while the noise branch is restricted to modeling only transient perturbations. Furthermore, we introduce an execution-time correction mechanism that leverages the learned structural representation to predict TD residuals, enabling lightweight and targeted value corrections without explicit planning. Experiments on the StarCraft Multi-Agent Challenge and Google Research Football benchmarks demonstrate superior sample efficiency and final performance compared to competitive baselines. Our code is available at: https://anonymous.4open.science/r/CDN-EE4B.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-Agent Reinforcement Learning, Decision making, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_itwhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SND, which separates agent representations into a Mamba-based structural branch and an MLP-based noise branch, then uses the structural representation to predict TD-residual corrections for top-k local Q-values. Experiments on SMAC and GRF show consistent performance gains.

Critical strengths

The method is technically coherent and computationally lightweight at execution time. The empirical evaluation is reasonably broad and includes ablations showing benefits from both disentanglement and TD correction.

Critical weaknesses

The claimed structure–noise disentanglement is not convincingly established. Orthogonality and adversarial return suppression only enforce representation specialization, but do not demonstrate that the two branches truly correspond to temporal structure and noise. The TD-correction mechanism also lacks a solid credit-assignment justification: every agent is supervised by the same team-level TD residual, which is not an agent-specific advantage signal, so it is unclear why the learned correction should reliably improve each agent’s action ranking.

Questions for authors

Can the authors provide controlled experiments showing that the noise branch specifically captures transient nuisance variation rather than arbitrary complementary information? Why should a shared team-level TD residual provide valid action-specific correction signals for individual agents?

4 comments
AuthorsOpenReview ↗

Response to Reviewer itwh 1/3

We sincerely thank the reviewer for the constructive comments. We further clarify the structure–noise representation learning and action-conditioned TD-residual correction mechanisms, together with their theoretical motivations and empirical evidence.

W1 & Q1. Structure–Noise Functional Specialization

  1. Clarification of the two branches. SND distinguishes the two representations according to their functional relevance to long-term returns and value estimation, rather than temporal duration alone.

    • Structural branch (zsz^s). Extracts history-dependent information useful for behavior modeling and value correction. It maintains additional temporal memory to aggregate cross-time dependencies and is supervised by structural behavior modeling and action-conditioned TD-residual prediction.
    • Noise branch (znz^n). Models behavioral variations not fully captured by the structural logits. Residual behavior modeling encourages complementary action prediction, while adversarial return supervision discourages return predictability from this representation.

    Here, complementary behavioral information describes the residual branch's role in behavior modeling, whereas nuisance variation concerns the relevance of information to the task objective. These notions are related but distinct. SND therefore does not treat all complementary information as nuisance; it further constrains the residual representation through adversarial return supervision.

    Likewise, transient variation refers to temporal duration, whereas nuisance variation concerns task relevance. An event occurring at a single timestep can still be structurally important if it affects subsequent transitions or returns.

    This task-oriented perspective is supported by prior representation-learning studies. Task Informed Abstractions (TIA; Fu et al., 2021) combines cooperative observation reconstruction with adversarial reward dissociation to separate reward-correlated features from distractors [1]. Denoised MDPs (Wang et al., 2022) further distinguish environmental information according to controllability and reward relevance [2]. SND adopts a related principle for multi-agent decision representations through residual behavior modeling and adversarial return supervision, while additionally using the structural representation for decentralized value correction.

  2. Predictive interpretation and architectural justification. For a fixed data distribution and a square-integrable return target GG, the optimal squared-error prediction risk satisfies

    \mathcal R^*(G\mid Z^n) &=\inf_g\mathbb E[(G-g(Z^n))^2]\\ &=\operatorname{Var}(G) -\operatorname{Var}(\mathbb E[G\mid Z^n]). \end{aligned}$$ This identity shows that, under an optimal predictor, increasing prediction risk discourages return-related variations that can be explained by the noise representation. Meanwhile, residual behavior modeling encourages $z^n$ to retain complementary behavioral information. Together, these objectives promote behavioral complementarity under a return-predictability constraint. Architecturally, both branches receive the same history-informed input, but the Mamba-based structural branch maintains additional temporal memory, whereas the MLP-based noise branch does not maintain additional recurrent state. Thus, temporally localized events are not automatically assigned to the noise branch; reward-relevant information can still be retained by the structural branch through its temporal representation and learning objectives.
AuthorsOpenReview ↗

Response to Reviewer itwh 2/3

  1. Controlled empirical evidence. The original submission included representation analyses in Appendices D.1 and D.4, which we have consolidated into Appendix D, Empirical Analysis of Structure–Noise Functional Specialization. In addition, we conducted a new controlled intervention study to examine the temporal responses of the two branches under persistent state changes and transient perceptual perturbations. We summarize the existing and newly added results below.

    (a) Frozen-representation probing and targeted objective ablations.

    We evaluate both frozen encoders on held-out MMM2 trajectories, using episode-level separation and identical standardized probes. Results are mean ± standard deviation over five independently trained seeds.

    AnalysisStructural zsz^sNoise znz^nTargeted ablation
    Timestep-controlled five-step return R2R^20.10 ± 0.040.01 ± 0.02w/o Ladv\mathcal L_{\mathrm{adv}}: 0.08 ± 0.03
    Action-conditioned TD-residual R2R^20.16 ± 0.050.04 ± 0.03—
    Residual action CE reduction—0.105 ± 0.028w/o Ln\mathcal L_n: 0.014 ± 0.011

    These results show that return-related and correction-relevant information is more readily decoded from zsz^s. Removing adversarial supervision increases the return predictability of znz^n, while removing residual behavior modeling substantially reduces its additional action-prediction contribution.

    (b) Simulator-level intervention analysis.

    On MMM2, we use three independently trained models and 24 paired episodes per seed. Enemy health is multiplied by 0.50.5 or 1.51.5, with a sham condition that leaves health unchanged. The clean, intervened, and sham environments share the same environment seed and identical pre-intervention actions, with zero pre-intervention observation discrepancies and action mismatches.

    Separate ridge probes predict intervention-induced future return changes from changes in either representation, using the same episode-grouped cross-validation protocol.

    Training seedStructural branchNoise branch
    Seed 10.780.25
    Seed 20.820.22
    Seed 30.830.19
    Mean0.810.22

    Entries are Pearson correlations between out-of-fold predictions and observed return changes. The structural representation consistently provides stronger predictions, supporting its preferential retention of information about the return consequences of the tested environmental changes. (c) Temporal response under matched-onset interventions.

We compare persistent enemy-health changes with single-frame perceptual perturbations on MMM2. Both conditions have identical observations at the intervention onset; thereafter, only the genuine state intervention continues to affect environment evolution.

We evaluate three independently trained, frozen SND checkpoints on 32 held-out episodes per seed. Surviving agents execute STOP, while eliminated agents execute NOOP. We measure the late-response energy ratio, defined as the normalized representation-change energy over steps 8–16 divided by that over steps 0–16. Results are mean ± standard deviation across three seeds.

InterventionStructural zsz^sNoise znz^n
Persistent health change44.0% ± 4.0%6.3% ± 1.0%
Single-frame perceptual perturbation17.0% ± 1.0%5.7% ± 2.0%

The structural branch exhibits a larger late response to persistent state changes, whereas the noise branch shows similarly low late-response ratios under both conditions, indicating partial temporal-scale specialization.

We further evaluate architecture-matched checkpoints without Lorth\mathcal L_{\mathrm{orth}}, using the interaction measure

T=(Ls,HP−Ln,HP)−(Ls,pulse−Ln,pulse),T=(L_{s,\mathrm{HP}}-L_{n,\mathrm{HP}}) -(L_{s,\mathrm{pulse}}-L_{n,\mathrm{pulse}}),

where LL denotes the late-response energy ratio.

ModelTT
Full SND0.26 ± 0.03
w/o Lorth\mathcal L_{\mathrm{orth}}0.07 ± 0.02

The interaction decreases across all three seeds without orthogonality regularization, supporting its contribution to the observed temporal-response specialization.

(d) Cross-branch dependence.

VariantExcess MI ↓Cross-branch CKA ↓
Full SND0.06 ± 0.020.14 ± 0.04
w/o Lorth\mathcal L_{\mathrm{orth}}0.23 ± 0.050.31 ± 0.08

The full model exhibits lower measured cross-branch dependence and representational similarity, complementing the predictive and intervention results.

Together, these analyses support stronger return-related information in the structural branch, complementary behavioral information under return-predictability constraints in the noise branch, and reduced representational redundancy between the two branches.

AuthorsOpenReview ↗

Response to Reviewer itwh 3/3

W2 & Q2. Action-Conditioned Correction under Shared Team-Level TD Supervision

1. Conditional-mean interpretation of the shared supervision. Equations (14)–(17) use the clipped team-level TD residual as the auxiliary prediction target. This target is not an identified individual advantage. However, for fixed base value estimates, representations, and data distribution, the squared-error prediction objective has the conditional-mean solution:

\mathbb{E}_{\mathcal{D}} \left[ \mathrm{clip}(\delta_{\mathrm{tot}},-c,c) \mid Z^s=z,\\ A=a \right].$$ Thus, sharing the supervision target does not require identical predictions across agents or candidate actions. The predictor estimates the component of the team-level value discrepancy that is predictable from each agent's local structural context and action. This is an action-conditioned regression signal rather than an identified individual advantage. Unlike counterfactual credit-assignment methods such as COMA, which construct an agent-specific advantage by marginalizing one agent's action while holding the other agents' actions fixed [3], SND uses shared team-level supervision to learn local action-conditioned corrections. 2. **Empirical evidence of action-value alignment.** Appendix C evaluates the directional alignment between predicted residual differences and estimated action-value gaps. We compare SND with a non-disentangled encoder on 3s5z and MMM2. | Encoder | Map | Sign correlation | Hard accuracy | Medium accuracy | Easy accuracy | |---|---|---:|---:|---:|---:| | Disentangled | 3s5z | **0.8293** | **0.7064** | **0.7867** | **0.8409** | | Disentangled | MMM2 | **0.8890** | **0.6825** | **0.7415** | **0.8625** | | Non-disentangled | 3s5z | 0.7685 | 0.6412 | 0.7248 | 0.8031 | | Non-disentangled | MMM2 | 0.8217 | 0.6189 | 0.6897 | 0.8264 | The disentangled encoder achieves stronger directional alignment and higher prediction accuracy across both maps and all decision-difficulty levels. In particular, sign correlation improves from 0.7685 to 0.8293 on 3s5z and from 0.8217 to 0.8890 on MMM2. These results indicate that conditioning the residual predictor on the structural representation and action yields directionally informative correction signals under the evaluated protocol, with stronger alignment when structure–noise disentanglement is used. 3. **Connection to the action-ranking analysis.** Appendix A provides a bias-aware analysis of how the learned correction affects action ranking. For a competing action $a$ and the reference preferred action $a^\dagger$, the expected corrected action margin is $$M_a=\Delta^\dagger(a)-b_a+\alpha m_a,$$ where $\Delta^\dagger(a)$ is the reference action-value gap, $b_a$ is the relative estimation bias, and $m_a$ is the expected correction-margin contribution. When $M_a>0$, the ranking-error probability admits the bound $$\Pr\left[ \widetilde Q_k(a)\geq \widetilde Q_k(a^\dagger) \mid \tau \right] \leq \frac{\sigma_a^2}{\sigma_a^2+M_a^2},$$ where $\sigma_a^2$ is the variance of the corrected action margin. This analysis characterizes how the expected correction margin and its variance affect ranking reliability. Together with the empirical alignment results and the performance improvements reported in the main experiments, it provides theoretical and empirical support for the proposed correction mechanism. Accordingly, SND uses an action-conditioned auxiliary predictor trained with shared team-level supervision, rather than an individual advantage estimator. ### References [1] X. Fu, G. Yang, P. Agrawal, and T. Jaakkola. "Learning Task Informed Abstractions." *Proceedings of the 38th International Conference on Machine Learning (ICML)*, PMLR 139, pp. 3480–3491, 2021. [2] T. Wang, S. Du, A. Torralba, P. Isola, A. Zhang, and Y. Tian. "Denoised MDPs: Learning World Models Better Than the World Itself." *Proceedings of the 39th International Conference on Machine Learning (ICML)*, PMLR 162, pp. 22591–22612, 2022. [3] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. "Counterfactual Multi-Agent Policy Gradients." *Proceedings of the AAAI Conference on Artificial Intelligence*, 32(1), pp. 2974–2982, 2018.

Dear Reviewer,

Thank you for your time, thoughtful assessment, and constructive feedback. We have now responded to each concern raised in your review and provided additional supporting evidence.

Should your schedule permit, we would be very grateful if you could review our responses and the newly added analyses at your convenience. We remain happy to clarify any outstanding questions.

Thank you once again for your time and consideration.

Reviewer_nyavOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new factorization approach in multi-agent reinforcement learning (MARL), which separates structural learning from coping with noise to facilitate more effective learning. It proposes SND, which uses two models, namely Mamba for the structural branch, and a simple MLP for the noise branch. Both branches are used to correct the temporal difference residual of common MARL methods for more accurate value learning. It is evaluated on SMAC and GRF and compared with a variety of state-of-the-art MARL methods.

Critical strengths

The paper introduces an interesting direction and machinery to address entanglement of temporal structure and noise in MARL.

Critical weaknesses

Insufficient benchmark choice

The choice of MARL benchmarks is insufficient for the claims of the paper, since SMAC and GRF are known to lack stochasticity [1,2,3] in contrast to simpler Dec-POMDP benchmarks [1,2,3,4]. Thus, it is unclear to me to what extent the noise entanglement is relevant here. The ablation in Figure 4 shows a difference in performance, but this could also be due to some interplay between the existing components. Thus, I recommend further investigation to understand the proposed approach fully.

Presentation of results

While the results suggest some improvement over the state-of-the-art (to fully assess its capability to cope with noise, I recommend more noisy benchmarks, as recommended in [2,3,4]), I would appreciate some insights about the disentanglement, e.g., a small toy problem like a matrix game, where one can clearly see how SND successfully captures the structures, while others approaches fail to do so. Without such a demonstration, the machinery just looks impressive on the equations.

Limitations

[1] Lyu et al., "A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning", AAAI 2022

[2] Phan et al., "Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability", ICML 2023

[3] Ellis et al., "SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning", NeurIPS Benchmarks 2023

[4] Oliehoek et al., "A Concise Introduction to Decentralized POMDPs", 2016

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Response to Reviewer nyav 1/3

W1. Benchmark stochasticity: additional evaluation on SMACv2

Thank you for highlighting the importance of benchmark stochasticity and recommending SMACv2. We have expanded the empirical evaluation to Protoss 5v5, Terran 5v5, and Zerg 5v5 in SMACv2, and added these results to the revised manuscript.

SMACv2 [1] procedurally randomizes unit compositions and starting positions across episodes. These changes introduce variability in the initial conditions and require agents to adapt to different combat configurations, providing an evaluation under the additional environmental stochasticity raised in your comment.

We compare SND with ACORM, R3DM, QMIX, and GoMARL under the same training and evaluation settings, using five random seeds for each method. The tables report win rates (%) as mean ± standard deviation at 1M and 2M environment steps. Bold entries indicate the highest mean in each column.

Protoss 5v5

Method1M environment steps2M environment steps
ACORM37.81 ± 3.9854.06 ± 1.33
R3DM40.94 ± 8.4056.51 ± 4.79
QMIX25.52 ± 4.5735.48 ± 0.20
GoMARL40.31 ± 3.2950.51 ± 7.09
SND53.42 ± 4.1162.79 ± 1.94

Terran 5v5

Method1M environment steps2M environment steps
ACORM51.33 ± 10.7261.22 ± 3.72
R3DM54.48 ± 7.2259.21 ± 3.30
QMIX39.71 ± 12.7044.96 ± 6.05
GoMARL44.00 ± 7.1159.88 ± 4.92
SND62.71 ± 10.3172.29 ± 1.30

Zerg 5v5

Method1M environment steps2M environment steps
ACORM28.12 ± 3.2040.00 ± 2.65
R3DM34.06 ± 7.5141.87 ± 0.88
QMIX13.59 ± 4.2026.24 ± 0.89
GoMARL27.62 ± 8.4843.88 ± 6.64
SND39.11 ± 4.0846.89 ± 4.29

SND also achieves higher mean win rates than the compared baselines when unit compositions and starting positions vary randomly across episodes. These experiments address your recommendation to evaluate the method under greater environmental stochasticity. The full learning curves and evaluation settings are now included in Figure 3 and Section 6.

References

[1] Ellis, B., et al. (2023). SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. NeurIPS Datasets and Benchmarks.

AuthorsOpenReview ↗

Response to Reviewer nyav 2/3

W2. Attribution of gains: disentanglement vs. correction

Thank you for highlighting the need to distinguish the contributions of disentanglement and TD correction. We examine component ablations and representation diagnostics to connect the performance differences to the roles of the two mechanisms.

  1. Disentanglement improves the residual signal used for correction. The component ablation compares full SND with TD correction (w/o disentangle), which applies top-k correction to a monolithic encoder. The reported curves on 3s5z_vs_3s6z and MMM2 show higher performance for the full model (Figure 5). Appendix C, Table 1 further compares the residual predictions of disentangled and non-disentangled encoders:

    MapEncoderSign correlationHard accuracyMedium accuracyEasy accuracy
    3s5zDisentangled0.82930.70640.78670.8409
    3s5zNon-disentangled0.76850.64120.72480.8031
    MMM2Disentangled0.88900.68250.74150.8625
    MMM2Non-disentangled0.82170.61890.68970.8264

    The disentangled encoder achieves better alignment with the estimated action-value-gap direction and higher prediction accuracy at every reported difficulty level, providing a concrete connection between separation and residual quality.

  2. Execution-time correction provides additional gains when disentanglement is retained. SND w/o top-k correction retains disentanglement but removes execution-time value correction; its lower performance in Figure 5 supports the contribution of applying the learned correction. Removing disentanglement or adversarial supervision also reduces performance in the reported component comparisons. Appendix D, Tables 4–5 examine whether the separation objectives produce their intended representation changes on MMM2 (mean ± SD over five training seeds):

    DiagnosticRemoved objectiveFull SNDAfter removal
    Noise-branch return-probe R2R^2Ladv\mathcal L_{\mathrm{adv}}0.01 ± 0.020.08 ± 0.03
    Additional action-prediction benefit ΔCE\Delta\mathrm{CE}Ln\mathcal L_n0.105 ± 0.0280.014 ± 0.011
    Cross-branch excess MILorth\mathcal L_{\mathrm{orth}}0.06 ± 0.020.23 ± 0.05
    Cross-branch CKALorth\mathcal L_{\mathrm{orth}}0.14 ± 0.040.31 ± 0.08

    These changes support the intended roles of suppressing return predictability in the noise branch, retaining complementary action information, and reducing cross-branch dependence.

These results support complementary roles for the two mechanisms: separation training improves the information used to predict residuals, and execution-time correction uses those predictions to adjust action values.

AuthorsOpenReview ↗

Response to Reviewer nyav 3/3

W3. Interpretable evaluation under stochastic observations

Thank you for suggesting a small, interpretable task with explicit stochasticity. We have added a two-agent Dec-Tiger experiment and inspected the learned policy to show how noisy observation histories affect decisions. Dec-Tiger is also used in the reviewer-cited study by Phan et al. [2] to evaluate stochastic partial observability.

  1. Dec-Tiger makes the hidden task state and observation uncertainty explicit. The tiger's location is hidden, and each agent receives an auditory observation with 85% accuracy when listening. Agents choose among listening, opening the left door, and opening the right door. We use H=4H=4 and γ=1\gamma=1; opening a door resets the hidden state, and the episode continues until the fixed horizon. Between door openings, the hidden state persists while auditory observations can disagree, making observation history useful for action selection.

  2. We evaluate all methods under a common training budget and report the final checkpoint. Each method uses 500,000 joint environment steps and five random seeds. For each trained policy, we compute the expected cumulative return by enumerating the environment's stochastic branches. The table reports the mean and sample standard deviation across training seeds on the original reward scale; bold marks the highest mean.

    MethodExpected cumulative return (mean ± SD)
    SND0.1317 ± 5.083
    Parameter-Matched−18.921 ± 13.733
    SND w/o Dis and TD−20.706 ± 22.427
    ACORM−10.434 ± 14.942
    QMIX−31.234 ± 14.244
    LAIES−11.211 ± 0.409
    GoMARL−27.500 ± 9.192
    RODE−8.000 ± 2.250
    VAST−34.000 ± 36.770
    R3DM−34.000 ± 20.560

    The two internal controls isolate different aspects of SND. Parameter-Matched retains the same two-branch architecture, parameter count, and TD correction, while removing orthogonality regularization and adversarial return constraints and allowing the residual action loss to update the structural branch. SND w/o Dis and TD removes both representation branches, their auxiliary losses, and TD correction, while retaining the base value learner, role representations. SND's mean return is higher than both controls under stochastic observations.

  3. The inspected policy changes its decision when an earlier observation changes, even though the current observation and previous actions are the same. For seed 2702, after listening in the first two steps, the policy chooses the following actions at step 3:

    Local observation historyCurrent observationSND action at step 3
    Left → LeftLeftOpen right
    Right → LeftLeftListen

    The timestep and the preceding listening actions are identical; the earlier observation differs. With a uniform prior, conditionally independent listening errors, and no intervening door opening, these histories correspond to Bayesian probabilities of 96.98% and 50% that the tiger is on the left. These probabilities are analytical reference values from the observation model. The policy example makes the use of earlier noisy evidence directly inspectable, complementing the return comparisons and representation diagnostics.

References

[2] Phan, T., et al. (2023). Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability. ICML, 27840–27853.

Dear Reviewer,

Thank you for your time, thoughtful assessment, and constructive feedback. We have now responded to each concern raised in your review and provided additional supporting evidence.

Should your schedule permit, we would be very grateful if you could review our responses and the newly added analyses at your convenience. We remain happy to clarify any outstanding questions.

Thank you once again for your time and consideration.

Reviewer_iLhkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a structure–noise disentanglement framework for separating reward-relevant temporal structure from noise in multi-agent reinforcement learning. The framework decomposes each agent’s representation into a structural branch intended to capture long-term coordination patterns and a noise branch that models instantaneous residuals. During execution, the learned structural representation is used to adjust Q-value estimates. The proposed method is evaluated on the SMAC and GRF benchmarks.

Critical strengths
  1. The idea of isolating the important environmental information from noise is important for efficient MARL.

  2. The method details are introduced fully and completely.

Critical weaknesses
  1. My primary concern is the lack of explicit mechanisms for inter-agent coordination. Both the temporal structure and noise are modeled separately for each agent, and the framework does not clearly explain how it captures inter-agent dependencies or facilitates coordinated behavior. The authors should clarify what makes the proposed approach specifically suited to MARL beyond applying a single-agent learning mechanism independently to multiple agents.

  2. The notion of “reward-relevant temporal structure” appears overly broad and insufficiently defined. Reward-relevant information may include agent positions, time, health status, actions, and pairwise distances, among other factors. It remains unclear why these heterogeneous variables should be grouped under the single notion of “reward-relevant temporal structure.”

  3. The experimental evaluation is limited and does not sufficiently establish the generality of the reported performance gains. Evaluations on additional benchmarks, such as SMACv2, would strengthen the empirical evidence.

  4. The presentation should be improved, particularly the ordering of definitions. In line 143, temporal structure and noise are defined using the history-informed shared input xx, but xx is not introduced until line 183. The authors should define xx before using it to formulate these concepts.

Questions for authors

Please refer to Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iLhk 1/3

W1. Inter-agent coordination and the MARL-specific design

Thank you for highlighting the need to clarify inter-agent coordination. Coordination in SND is implemented through the underlying CTDE learner's joint value optimization and SND's shared team-level TD supervision of local corrections. All agents' utilities contribute to the same team value estimate, whose TD error couples their training; SND further uses this team-level error to train each agent's structural representation and correction head.

  1. Agents’ local utilities are jointly optimized through the team value function. The underlying CTDE learner combines agents' local utilities into a joint action value and trains them using the shared team reward, following the value-decomposition approach exemplified by VDN [1], QMIX [2], QTRAN [3], and QPLEX [4].

    =f_{\mathrm{mix}}\!\left(Q_1(\tau_1^t,a_1^t),\ldots, Q_N(\tau_N^t,a_N^t);s^t\right).$$ For an individual training sample, the TD loss is $\ell_{\mathrm{RL}}^t = \left(Q_{\mathrm{tot}}^t - \operatorname{sg}(y^t)\right)^2$. Defining the team-level TD residual as $\delta_{\mathrm{tot}}^t = y^t - Q_{\mathrm{tot}}^t$, the gradient received by each agent's local utility is $\frac{\partial \ell_{\mathrm{RL}}^t}{\partial Q_i} = -2\delta_{\mathrm{tot}}^t\frac{\partial Q_{\mathrm{tot}}^t}{\partial Q_i}$. Each local utility is updated through the mixing network using a TD error computed from the joint action value of all agents. Consequently, changing another agent's utility can change the TD error used to update this agent.
  2. SND further trains each agent’s temporal representation and local correction using team-level value feedback. Specifically, each agent predicts the same team-level TD residual, constructed from the team reward and joint value estimates, using its own structural representation and executed action (Eqs. 14–17).

    =y^t-Q_{\mathrm{tot}}^{\mathrm{eval}} \!\left(\boldsymbol{\tau}^t,\mathbf a^t,s^t\right), \qquad \hat\delta_i^t =g_{\mathrm{td}}\!\left([z_i^{s,t};\operatorname{onehot}(a_i^t)]\right).$$ Here, $y^t$ is the one-step TD target computed from the team reward and the target joint value. This objective trains the structural encoder and correction head and also contributes gradients to the base utility, mixing, and role-representation networks. Thus, agents' local representations and corrections are trained against a shared measure of the team's value-estimation error.
  3. SND turns team-level training feedback into locally computable action corrections. At execution, each agent uses its local history to predict action-conditioned corrections and adjust its action values (Eqs. 19–22), preserving decentralized execution under CTDE.

SMAC and GRF cover coordinated combat, offensive cooperation, and long-horizon team interactions. SND achieves improved performance on these tasks (Figs. 2 - 4). The ablations further show that removing disentanglement or TD-residual correction degrades performance (Fig. 5). Moreover, Appendix C shows strong directional alignment between predicted residuals and action-value gaps. Together, these results support the effectiveness of SND in improving decentralized cooperative decision-making. We have clarified this coordination pathway in Section 5.5.

References

[1] Sunehag, P., et al. (2018). Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. AAMAS, 2085–2087.

[2] Rashid, T., et al. (2020). Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. JMLR, 21(178), 1–51.

[3] Son, K., et al. (2019). QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning. ICML, 5887–5896.

[4] Wang, J., et al. (2021). QPLEX: Duplex Dueling Multi-Agent Q-Learning. ICLR.

AuthorsOpenReview ↗

Response to Reviewer iLhk 2/3

W2. Definition of reward-relevant temporal structure

Thank you for pointing out that “reward-relevant temporal structure” needs a clearer definition. In SND, reward-relevant temporal structure refers to information from interaction history that helps predict future team returns and evaluate actions.

  1. Information is grouped by its usefulness for return prediction and action evaluation, rather than by observation type. Positions, health, actions, and pairwise distances may each carry such information; their relevance depends on the task and interaction history. Here, temporal refers to the use of interaction history, including rapidly changing information when it is reward-relevant. Defining task relevance through rewards and control is a common perspective in RL representation learning [5] [6].

  2. SND learns this functional distinction through predictive objectives applied to the same input, rather than assigning observation types to different branches. Both branches receive the same history-informed input (Eq. 4).

    x_i^t&=[\operatorname{sg}(b_i^t);\operatorname{sg}(r_i^t);\tilde a_i^{t-1}],\\ z_i^{s,t}&=f_s(x_i^{1:t}),\qquad z_i^{n,t}=f_n(x_i^t). \end{aligned}$$ The structural branch uses Mamba to aggregate information across time and is trained through behavior prediction and action-conditioned TD-residual prediction. The noise branch models complementary behavioral variations, while adversarial return supervision suppresses its return predictability.
  3. The held-out analyses show higher return predictability in the structural branch and complementary action information in the noise branch. On MMM2, with frozen encoders and episode-disjoint probe-training and test sets, the structural representation yields higher test R2R^2 for both targets below. Results are mean ± standard deviation over five independently trained seeds (Appendix D).

    Probe targetStructural representationNoise representation
    Timestep-controlled five-step return0.10 ± 0.040.01 ± 0.02
    Action-conditioned TD residual0.16 ± 0.050.04 ± 0.03

    The noise branch contributes complementary action information, reducing action-reconstruction cross-entropy by 0.105±0.0280.105 \pm 0.028. Removing adversarial supervision increases its return-probe R2R^2 from 0.01±0.020.01 \pm 0.02 to 0.08±0.030.08 \pm 0.03.

    Under controlled health interventions, changes in the structural representation also predict return changes more strongly: the Pearson correlations are 0.780 and 0.820 across two seeds, compared with 0.250 and 0.220 for the noise representation (Appendix D). We have clarified the functional definition at the beginning of Section 4 and revised the abstract to use consistent terminology.

References

[5] Fu, X., et al. (2021). Learning Task Informed Abstractions. ICML.

[6] Wang, T., et al. (2022). Denoised MDPs: Learning World Models Better Than the World Itself. ICML.

AuthorsOpenReview ↗

Response to Reviewer iLhk 3/3

W3. Additional evaluation on SMACv2

Thank you for suggesting SMACv2. We have extended the evaluation to Protoss 5v5, Terran 5v5, and Zerg 5v5, comparing SND with ACORM, R3DM, QMIX, and GoMARL. SMACv2 introduces variation in unit compositions and starting positions [7], providing an additional test of performance across diverse cooperative scenarios.

All methods use the same training and evaluation settings, with five random seeds each. The tables report win rates (%) as mean ± standard deviation at 1 million and 2 million environment steps. Bold entries indicate the highest mean in each column.

Protoss 5v5

Method1M environment steps2M environment steps
ACORM37.81 ± 3.9854.06 ± 1.33
R3DM40.94 ± 8.4056.51 ± 4.79
QMIX25.52 ± 4.5735.48 ± 0.20
GoMARL40.31 ± 3.2950.51 ± 7.09
SND53.42 ± 4.1162.79 ± 1.94

Terran 5v5

Method1M environment steps2M environment steps
ACORM51.33 ± 10.7261.22 ± 3.72
R3DM54.48 ± 7.2259.21 ± 3.30
QMIX39.71 ± 12.7044.96 ± 6.05
GoMARL44.00 ± 7.1159.88 ± 4.92
SND62.71 ± 10.3172.29 ± 1.30

Zerg 5v5

Method1M environment steps2M environment steps
ACORM28.12 ± 2.7040.00 ± 2.65
R3DM34.06 ± 7.5141.87 ± 0.88
QMIX13.59 ± 4.2026.24 ± 0.89
GoMARL27.62 ± 8.4843.88 ± 6.64
SND39.11 ± 4.0846.89 ± 4.29

SND achieves the highest mean win rate in all three SMACv2 scenarios at both 1M and 2M steps. These results extend the empirical support for SND from SMAC and GRF to SMACv2. We have added the full learning curves in Figure 3 and described the evaluation protocol in Section 6.

W4. Ordering of definitions

Thank you for pointing out this ordering issue. We have revised Section 4 to introduce the history-informed shared input xitx_i^t before presenting the structure–noise factorization and added a cross-reference to Section 5 for its detailed construction. We have also defined xi1:tx_i^{1:t} immediately after the factorization. These changes are highlighted in blue.

References

[7] Ellis, B., et al. (2023). SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. NeurIPS Datasets and Benchmarks.

Dear Reviewer,

Thank you for your time, thoughtful assessment, and constructive feedback. We have now responded to each concern raised in your review and provided additional supporting evidence.

Should your schedule permit, we would be very grateful if you could review our responses and the newly added analyses at your convenience. We remain happy to clarify any outstanding questions.

Thank you once again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.