- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DEAL, which combines local next-observation prediction with residual-based critic signal reweighting to reduce the CTDE decentralized-execution gap. Experiments on 37 tasks across four MARL benchmarks show strong gains over CTDE and CTCE baselines.
Critical strengths
The empirical evaluation is broad and statistically solid, with 10 seeds, multiple benchmark suites, strong baselines, and targeted ablations. The method performs particularly well under severe partial observability, which is consistent with its intended motivation.
Critical weaknesses
The novelty is limited: next-observation prediction and uncertainty-weighted learning are both established ideas, and their combination is somewhat incremental. The residual-based weighting is insufficiently justified. A large return residual may reflect useful hard samples rather than unreliable critic signals, so downweighting them can introduce biased policy updates without theoretical guarantees.
Questions for authors
What objective is actually optimized by the residual-weighted PPO update, and under what assumptions does it preserve the original policy optimum? Can the authors show that the gains come specifically from “execution alignment” rather than generic auxiliary representation learning?
3 comments
-
Bounds on deviations.
The optimum is not preserved in general, and the reason is specific to CTDE. The weights use the critic's centralized input, which the actor does not see. Reweighting situations that the actor can tell apart only changes how fast each is learned, which with shared parameters acts like any other change of the state distribution. A preference can change only when situations that the actor cannot tell apart receive different weights and favor different actions. We bound this effect in three ways.
- Within a local history. The change of an action-value gap equals , where is the context hidden from the actor (Proposition 2). The estimated average bound on RWARE is about 12% of a typical gap (Appendix D.4).
- Margin condition. A preference whose margin exceeds this bound cannot be reversed.
- Bounded suboptimality for any policy class (Theorem 1). Within the trust region, the maximizer of the weighted surrogate loses at most on the PPO surrogate, so the loss shrinks in proportion to the spread of the weights. Here is the standard deviation of the weights, is the clipping range, and is the decision signal. Since the expected update changes by about 10% of its norm on trained agents (Figure 11b), the weighted update also remains an ascent direction of the PPO surrogate (Proposition 1).
The weighting rule itself balances these bounds against noise reduction. It minimizes a proxy of the gradient second moment in Proposition 1 plus a penalty on the spread of the weights, and this spread sets all the bounds above (Appendix A.3).
These results concern the surrogate of each update, the level at which PPO itself is usually analyzed. We do not claim a guarantee on the optimal policy of the underlying task. We thank the reviewer for this question, which led us to make the objective of DEAL precise.
3. Does reliability weighting suppress useful learning signals? (W2)
We agree that a large residual can reflect both useful action-value differences and noise. Our new analyses show that DEAL reduces gradient noise while largely preserving useful learning signals. We added empirical gradient diagnostics (Appendix D.4 and Section 4.3) and theoretical analysis (Appendix A.5).
-
Informative learning signals are preserved while gradient noise is reduced.
On RWARE, samples before team rewards or with the strongest decision signals receive weights similar to other samples. Samples before reward events retain 98% of their gradient contribution on RWARE and 93% on SMAX. The estimated action-dependent part accounts for less than 1% of the advantage variance across all predicted residual levels, while the variability within each action grows with the predicted residual (Figure 11c).
DEAL reduces policy-gradient variance by 3–10% on RWARE and about 29% early in training on SMAX, while shuffling the same weights increases variance. Our theory also shows that the weighting can change the update only where the weights depart from one and the agent's own action matters (Proposition 1 and Theorem 1). These are the hard samples you raised, and they keep weights similar to other samples. Together, these results suggest that DEAL preserves useful learning signals while reducing gradient noise.
-
The way the weights are computed matters.
Table C compares different weighting methods while keeping the predictive representation fixed. The new variants were tuned per task, except for the local-information control, which uses the same hyperparameters as the predictive-only variant.
Table C. Weights Return (95% CI) No weighting (predictive only) 47.4 [43.0, 51.9] From the actor's local information (new) 46.9 [42.1, 51.7] From each sample's realized advantage (new) 38.3 [35.1, 41.4] From the critic's features, with action-dependent correction (new) 57.9 [53.0, 62.7] From the critic's features (DEAL) 65.4 [61.3, 69.4] Removing the action-dependent part of the residual does not improve performance. We introduced a new variant that estimates and removes this component from the residual target. However, it performs below DEAL (Table C). Because the learned correction may introduce noise, we treat this as supporting evidence alongside the diagnostics.
Weights from the realized advantage are harmful. They depend on the sampled action and can change which actions the policy update favors. They also lower the return below that of no weighting. Weights predicted from the actor's local information provide no improvement either, supporting the importance of centralized information in CTDE. A probe also recovers DEAL's weights from the critic's features but much less from the actor's recurrent state (Appendix D.4).
4. Do the gains come from execution alignment rather than generic auxiliary representation learning? (Q2)
By execution alignment, we mean designing training signals around the information available to each agent during decentralized execution. A generic auxiliary effect would help regardless of what is predicted. Our results show the opposite. The gain depends strongly on the target and is largest for the agent's own local future.
Table D changes only the prediction target, with the same backbone, weighting, and training budget. The first row has no auxiliary prediction at all.
| Table D. Prediction target | Return | Gain over no prediction |
|---|---|---|
| No auxiliary prediction (weighting only) | 41.6 [38.7, 44.3] | |
| Past joint trajectory | 20.7 [18.6, 22.8] | −20.9 |
| Next global state | 42.5 [39.0, 45.9] | +0.9 |
| Current global state | 49.0 [45.5, 52.4] | +7.5 |
| Own current observation (new) | 51.6 [47.6, 55.5] | +10.0 |
| Own next observation (DEAL) | 65.4 [61.3, 69.4] | +23.8 |
The three centralized targets are common auxiliary tasks, yet none comes close to DEAL. The new variant reconstructs the agent's current observation instead of predicting its next observation, with all other components unchanged. It gives less than half of DEAL's gain.
-
Centralized targets degrade as the unobserved information grows.
In the LBF sight sweep (Section 4.3 and Figure 6c), DEAL's improvement grows as the observation range shrinks. To further examine the effect of limited local information, we conducted an analysis of RWARE tasks with more agents on the same map, where a local history determines less of the centralized information. Table E (Table 7 in the revised paper) gives the return of each variant divided by that of DEAL, for all five pairs in RWARE.
Table E. Prediction target tiny2 → 4tiny-hard2 → 4medium4 → 6large4 → 8large-hard4 → 8Average Next global state 0.93 → 0.82 0.89 → 0.88 0.68 → 0.36 0.71 → 0.20 0.52 → 0.32 0.74 → 0.52 Current global state 1.04 → 0.89 0.83 → 0.84 0.73 → 0.37 0.83 → 0.33 0.99 → 0.47 0.88 → 0.58 Past joint trajectory 1.05 → 0.07 0.96 → 0.28 0.11 → 0.11 0.51 → 0.08 0.18 → 0.09 0.56 → 0.13 Own current observation 0.99 → 0.92 0.82 → 0.88 0.81 → 0.72 0.57 → 0.64 0.74 → 0.77 0.79 → 0.79 On average, all three centralized targets fall when agents are added, while the local target does not change.
These results show that the gain comes from aligning the prediction target with what the agent observes at execution, and not from auxiliary learning in general. If the reviewer meant something different by execution alignment, please let us know, and we will address that reading directly.
Thank you again for your careful review and constructive questions. We believe DEAL indeed provides a useful framework to reduce the decentralized execution gap for widely-considered CTDE. We would be happy to discuss any further questions.
Thank you for the careful review and constructive comments. We here address the reviewer's comments: the novelty first, then the weighted objective, useful learning signals, and execution alignment. Please also see our Common Response for the summary of our responses to all reviewers, and all changes are marked in red in the revised paper.
1. Regarding the novelty of the work (W1)
We agree that next-observation prediction and uncertainty-weighted learning each have a history. DEAL designs these two components specifically for CTDE and shows that they address two distinct training-execution mismatches. The paper makes three contributions, summarized below:
In CTDE, a local target can be the more effective.
DEAL shapes a representation with an auxiliary prediction loss, as world models and self-predictive representations do. What differs is the target and the purpose. In this paper, we focus on what makes decentralized execution difficult. In partial observability, unseen teammates appear as several possible local futures, so we model the prediction target with a mixture and give its predictive scale to the policy at execution.
To our knowledge, DEAL is the first in CTDE to show, under the same backbone, that a purely local prediction target can outperform centralized targets when the predictive model is properly structured (Table 2 and 4. Q2 below). We clarified these distinctions and strengthened the related work of the revised paper.
The centralized critic can improve policy learning through reliability weighting.
Prior uncertainty-weighting methods reduce unreliable learning signals by weighting uncertain value targets or out-of-distribution actions, or by adjusting a global trust region. DEAL weights the policy-gradient samples of decentralized actors, with an estimate predicted from the centralized features. During the rebuttal, we added theoretical and empirical analyses to explain how the weighting improves policy learning (Appendices A.5 and D.4, and 2. Q1 and 3. W2 below).
The two parts comprise one framework and reinforce each other.
Partial observability makes an agent's next local observation uncertain. This also causes returns to vary across samples that look similar to the actor. DEAL addresses these problems through predictive representation and critic-based reliability weighting.
Below, Table A shows the component results from Table 2. Table B shows the share of successful runs and their average return over 15 RWARE tasks with 10 seeds each.
The predictive representation improves the return of successful runs, while reliability weighting mainly increases how often learning succeeds. Together, they improve both outcomes, producing a gain greater than the sum of their individual improvements. These results show that the two components are not simply separate additions, but reinforce each other within a unified framework.
2. What objective does the weighted PPO update optimize, and when is the original policy optimum preserved? (Q1)
The weighted update optimizes the standard PPO surrogate under a reweighted rollout distribution, and we characterize when the original PPO optimum is preserved and how far it can move otherwise. These results are new in the revised paper (Appendix A.5, summarized in Section 3.2). We briefly describe them below.
Optimizing objective.
Let x be the critic input, ν(x) its rollout distribution, and w(x) the positive weight with mean one. DEAL optimizes the PPO surrogate under νw(x)=w(x)ν(x) (Lemma 4). Since w is positive and does not depend on the action, min(ρwA^,clip(ρ)wA^)=wmin(ρA^,clip(ρ)A^). The action distribution and the clipping of every sample are therefore unchanged.
Conditions for preserving the optimum.
We regard the original optimum as the actions that the unweighted PPO update prefers at each local history. These preferences are preserved whenever the weights are uncorrelated with the action-value gaps across the situations that share a local history (Proposition 2). This holds under full observability, and whenever the hidden situation shifts the return of all actions by the same amount.