- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SND, which separates agent representations into a Mamba-based structural branch and an MLP-based noise branch, then uses the structural representation to predict TD-residual corrections for top-k local Q-values. Experiments on SMAC and GRF show consistent performance gains.
Critical strengths
The method is technically coherent and computationally lightweight at execution time. The empirical evaluation is reasonably broad and includes ablations showing benefits from both disentanglement and TD correction.
Critical weaknesses
The claimed structure–noise disentanglement is not convincingly established. Orthogonality and adversarial return suppression only enforce representation specialization, but do not demonstrate that the two branches truly correspond to temporal structure and noise. The TD-correction mechanism also lacks a solid credit-assignment justification: every agent is supervised by the same team-level TD residual, which is not an agent-specific advantage signal, so it is unclear why the learned correction should reliably improve each agent’s action ranking.
Questions for authors
Can the authors provide controlled experiments showing that the noise branch specifically captures transient nuisance variation rather than arbitrary complementary information? Why should a shared team-level TD residual provide valid action-specific correction signals for individual agents?
4 comments
Response to Reviewer itwh 2/3
-
Controlled empirical evidence. The original submission included representation analyses in Appendices D.1 and D.4, which we have consolidated into Appendix D, Empirical Analysis of Structure–Noise Functional Specialization. In addition, we conducted a new controlled intervention study to examine the temporal responses of the two branches under persistent state changes and transient perceptual perturbations. We summarize the existing and newly added results below.
(a) Frozen-representation probing and targeted objective ablations.
We evaluate both frozen encoders on held-out MMM2 trajectories, using episode-level separation and identical standardized probes. Results are mean ± standard deviation over five independently trained seeds.
Analysis Structural Noise Targeted ablation Timestep-controlled five-step return 0.10 ± 0.04 0.01 ± 0.02 w/o : 0.08 ± 0.03 Action-conditioned TD-residual 0.16 ± 0.05 0.04 ± 0.03 — Residual action CE reduction — 0.105 ± 0.028 w/o : 0.014 ± 0.011 These results show that return-related and correction-relevant information is more readily decoded from . Removing adversarial supervision increases the return predictability of , while removing residual behavior modeling substantially reduces its additional action-prediction contribution.
(b) Simulator-level intervention analysis.
On MMM2, we use three independently trained models and 24 paired episodes per seed. Enemy health is multiplied by or , with a sham condition that leaves health unchanged. The clean, intervened, and sham environments share the same environment seed and identical pre-intervention actions, with zero pre-intervention observation discrepancies and action mismatches.
Separate ridge probes predict intervention-induced future return changes from changes in either representation, using the same episode-grouped cross-validation protocol.
Training seed Structural branch Noise branch Seed 1 0.78 0.25 Seed 2 0.82 0.22 Seed 3 0.83 0.19 Mean 0.81 0.22 Entries are Pearson correlations between out-of-fold predictions and observed return changes. The structural representation consistently provides stronger predictions, supporting its preferential retention of information about the return consequences of the tested environmental changes. (c) Temporal response under matched-onset interventions.
We compare persistent enemy-health changes with single-frame perceptual perturbations on MMM2. Both conditions have identical observations at the intervention onset; thereafter, only the genuine state intervention continues to affect environment evolution.
We evaluate three independently trained, frozen SND checkpoints on 32 held-out episodes per seed. Surviving agents execute STOP, while eliminated agents execute NOOP. We measure the late-response energy ratio, defined as the normalized representation-change energy over steps 8–16 divided by that over steps 0–16. Results are mean ± standard deviation across three seeds.
| Intervention | Structural | Noise |
|---|---|---|
| Persistent health change | 44.0% ± 4.0% | 6.3% ± 1.0% |
| Single-frame perceptual perturbation | 17.0% ± 1.0% | 5.7% ± 2.0% |
The structural branch exhibits a larger late response to persistent state changes, whereas the noise branch shows similarly low late-response ratios under both conditions, indicating partial temporal-scale specialization.
We further evaluate architecture-matched checkpoints without , using the interaction measure
where denotes the late-response energy ratio.
| Model | |
|---|---|
| Full SND | 0.26 ± 0.03 |
| w/o | 0.07 ± 0.02 |
The interaction decreases across all three seeds without orthogonality regularization, supporting its contribution to the observed temporal-response specialization.
(d) Cross-branch dependence.
| Variant | Excess MI ↓ | Cross-branch CKA ↓ |
|---|---|---|
| Full SND | 0.06 ± 0.02 | 0.14 ± 0.04 |
| w/o | 0.23 ± 0.05 | 0.31 ± 0.08 |
The full model exhibits lower measured cross-branch dependence and representational similarity, complementing the predictive and intervention results.
Together, these analyses support stronger return-related information in the structural branch, complementary behavioral information under return-predictability constraints in the noise branch, and reduced representational redundancy between the two branches.
Response to Reviewer itwh 3/3
W2 & Q2. Action-Conditioned Correction under Shared Team-Level TD Supervision
1. Conditional-mean interpretation of the shared supervision. Equations (14)–(17) use the clipped team-level TD residual as the auxiliary prediction target. This target is not an identified individual advantage. However, for fixed base value estimates, representations, and data distribution, the squared-error prediction objective has the conditional-mean solution:
\mathbb{E}_{\mathcal{D}} \left[ \mathrm{clip}(\delta_{\mathrm{tot}},-c,c) \mid Z^s=z,\\ A=a \right].$$ Thus, sharing the supervision target does not require identical predictions across agents or candidate actions. The predictor estimates the component of the team-level value discrepancy that is predictable from each agent's local structural context and action. This is an action-conditioned regression signal rather than an identified individual advantage. Unlike counterfactual credit-assignment methods such as COMA, which construct an agent-specific advantage by marginalizing one agent's action while holding the other agents' actions fixed [3], SND uses shared team-level supervision to learn local action-conditioned corrections. 2. **Empirical evidence of action-value alignment.** Appendix C evaluates the directional alignment between predicted residual differences and estimated action-value gaps. We compare SND with a non-disentangled encoder on 3s5z and MMM2. | Encoder | Map | Sign correlation | Hard accuracy | Medium accuracy | Easy accuracy | |---|---|---:|---:|---:|---:| | Disentangled | 3s5z | **0.8293** | **0.7064** | **0.7867** | **0.8409** | | Disentangled | MMM2 | **0.8890** | **0.6825** | **0.7415** | **0.8625** | | Non-disentangled | 3s5z | 0.7685 | 0.6412 | 0.7248 | 0.8031 | | Non-disentangled | MMM2 | 0.8217 | 0.6189 | 0.6897 | 0.8264 | The disentangled encoder achieves stronger directional alignment and higher prediction accuracy across both maps and all decision-difficulty levels. In particular, sign correlation improves from 0.7685 to 0.8293 on 3s5z and from 0.8217 to 0.8890 on MMM2. These results indicate that conditioning the residual predictor on the structural representation and action yields directionally informative correction signals under the evaluated protocol, with stronger alignment when structure–noise disentanglement is used. 3. **Connection to the action-ranking analysis.** Appendix A provides a bias-aware analysis of how the learned correction affects action ranking. For a competing action $a$ and the reference preferred action $a^\dagger$, the expected corrected action margin is $$M_a=\Delta^\dagger(a)-b_a+\alpha m_a,$$ where $\Delta^\dagger(a)$ is the reference action-value gap, $b_a$ is the relative estimation bias, and $m_a$ is the expected correction-margin contribution. When $M_a>0$, the ranking-error probability admits the bound $$\Pr\left[ \widetilde Q_k(a)\geq \widetilde Q_k(a^\dagger) \mid \tau \right] \leq \frac{\sigma_a^2}{\sigma_a^2+M_a^2},$$ where $\sigma_a^2$ is the variance of the corrected action margin. This analysis characterizes how the expected correction margin and its variance affect ranking reliability. Together with the empirical alignment results and the performance improvements reported in the main experiments, it provides theoretical and empirical support for the proposed correction mechanism. Accordingly, SND uses an action-conditioned auxiliary predictor trained with shared team-level supervision, rather than an individual advantage estimator. ### References [1] X. Fu, G. Yang, P. Agrawal, and T. Jaakkola. "Learning Task Informed Abstractions." *Proceedings of the 38th International Conference on Machine Learning (ICML)*, PMLR 139, pp. 3480–3491, 2021. [2] T. Wang, S. Du, A. Torralba, P. Isola, A. Zhang, and Y. Tian. "Denoised MDPs: Learning World Models Better Than the World Itself." *Proceedings of the 39th International Conference on Machine Learning (ICML)*, PMLR 162, pp. 22591–22612, 2022. [3] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. "Counterfactual Multi-Agent Policy Gradients." *Proceedings of the AAAI Conference on Artificial Intelligence*, 32(1), pp. 2974–2982, 2018.Dear Reviewer,
Thank you for your time, thoughtful assessment, and constructive feedback. We have now responded to each concern raised in your review and provided additional supporting evidence.
Should your schedule permit, we would be very grateful if you could review our responses and the newly added analyses at your convenience. We remain happy to clarify any outstanding questions.
Thank you once again for your time and consideration.
Response to Reviewer itwh 1/3
We sincerely thank the reviewer for the constructive comments. We further clarify the structure–noise representation learning and action-conditioned TD-residual correction mechanisms, together with their theoretical motivations and empirical evidence.
W1 & Q1. Structure–Noise Functional Specialization
Clarification of the two branches. SND distinguishes the two representations according to their functional relevance to long-term returns and value estimation, rather than temporal duration alone.
Here, complementary behavioral information describes the residual branch's role in behavior modeling, whereas nuisance variation concerns the relevance of information to the task objective. These notions are related but distinct. SND therefore does not treat all complementary information as nuisance; it further constrains the residual representation through adversarial return supervision.
Likewise, transient variation refers to temporal duration, whereas nuisance variation concerns task relevance. An event occurring at a single timestep can still be structurally important if it affects subsequent transitions or returns.
This task-oriented perspective is supported by prior representation-learning studies. Task Informed Abstractions (TIA; Fu et al., 2021) combines cooperative observation reconstruction with adversarial reward dissociation to separate reward-correlated features from distractors [1]. Denoised MDPs (Wang et al., 2022) further distinguish environmental information according to controllability and reward relevance [2]. SND adopts a related principle for multi-agent decision representations through residual behavior modeling and adversarial return supervision, while additionally using the structural representation for decentralized value correction.
Predictive interpretation and architectural justification. For a fixed data distribution and a square-integrable return target G, the optimal squared-error prediction risk satisfies
\mathcal R^*(G\mid Z^n) &=\inf_g\mathbb E[(G-g(Z^n))^2]\\ &=\operatorname{Var}(G) -\operatorname{Var}(\mathbb E[G\mid Z^n]). \end{aligned}$$ This identity shows that, under an optimal predictor, increasing prediction risk discourages return-related variations that can be explained by the noise representation. Meanwhile, residual behavior modeling encourages $z^n$ to retain complementary behavioral information. Together, these objectives promote behavioral complementarity under a return-predictability constraint. Architecturally, both branches receive the same history-informed input, but the Mamba-based structural branch maintains additional temporal memory, whereas the MLP-based noise branch does not maintain additional recurrent state. Thus, temporally localized events are not automatically assigned to the noise branch; reward-relevant information can still be retained by the structural branch through its temporal representation and learning objectives.