acceptodds
Under review as a conference paper at ICLR 2027

Bridging the Decentralized-Execution Gap in Multi-Agent Reinforcement Learning

Abstract

Centralized training with decentralized execution (CTDE) is the standard paradigm for cooperative multi-agent reinforcement learning, but partial observability can lead to a substantial performance gap compared to fully centralized approaches. We identify two training-execution mismatches in standard CTDE: Actors are trained with signals informed by centralized information that is unavailable at decentralized execution, while policy updates ignore the fact that the centralized critic signal's reliability can vary across agent-and-time samples under partial observability and decentralized execution. We introduce Decentralized-Execution-Aligned Learning (DEAL) to address these mismatches. DEAL uses a *decentralized predictive representation* to ground actor-side learning by training each recurrent state to predict the agent's next observation using a conditional mixture model capturing the ambiguity induced by hidden context associated with decentralized operation. DEAL further uses *reliability-weighted critic signals* to predict the collection-time residual scale from critic features and adjust each sample's relative influence on the policy update. Empirically, across 37 tasks from four diverse environments, DEAL attains the best performance in every environment, outperforming strong CTDE and CTCE baselines under the same local policy model complexity budget.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, centralized training with decentralized execution, partial observability, predictive representation learning, policy optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2m4TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DEAL, which combines local next-observation prediction with residual-based critic signal reweighting to reduce the CTDE decentralized-execution gap. Experiments on 37 tasks across four MARL benchmarks show strong gains over CTDE and CTCE baselines.

Critical strengths

The empirical evaluation is broad and statistically solid, with 10 seeds, multiple benchmark suites, strong baselines, and targeted ablations. The method performs particularly well under severe partial observability, which is consistent with its intended motivation.

Critical weaknesses

The novelty is limited: next-observation prediction and uncertainty-weighted learning are both established ideas, and their combination is somewhat incremental. The residual-based weighting is insufficiently justified. A large return residual may reflect useful hard samples rather than unreliable critic signals, so downweighting them can introduce biased policy updates without theoretical guarantees.

Questions for authors

What objective is actually optimized by the residual-weighted PPO update, and under what assumptions does it preserve the original policy optimum? Can the authors show that the gains come specifically from “execution alignment” rather than generic auxiliary representation learning?

3 comments

Thank you for the careful review and constructive comments. We here address the reviewer's comments: the novelty first, then the weighted objective, useful learning signals, and execution alignment. Please also see our Common Response for the summary of our responses to all reviewers, and all changes are marked in red{\color{RubineRed}\text{red}} in the revised paper.


1. Regarding the novelty of the work (W1)

We agree that next-observation prediction and uncertainty-weighted learning each have a history. DEAL designs these two components specifically for CTDE and shows that they address two distinct training-execution mismatches. The paper makes three contributions, summarized below:


  • In CTDE, a local target can be the more effective.

    DEAL shapes a representation with an auxiliary prediction loss, as world models and self-predictive representations do. What differs is the target and the purpose. In this paper, we focus on what makes decentralized execution difficult. In partial observability, unseen teammates appear as several possible local futures, so we model the prediction target with a mixture and give its predictive scale to the policy at execution.

    To our knowledge, DEAL is the first in CTDE to show, under the same backbone, that a purely local prediction target can outperform centralized targets when the predictive model is properly structured (Table 2 and 4. Q2 below). We clarified these distinctions and strengthened the related work of the revised paper.


  • The centralized critic can improve policy learning through reliability weighting.

    Prior uncertainty-weighting methods reduce unreliable learning signals by weighting uncertain value targets or out-of-distribution actions, or by adjusting a global trust region. DEAL weights the policy-gradient samples of decentralized actors, with an estimate predicted from the centralized features. During the rebuttal, we added theoretical and empirical analyses to explain how the weighting improves policy learning (Appendices A.5 and D.4, and 2. Q1 and 3. W2 below).


  • The two parts comprise one framework and reinforce each other.

    Partial observability makes an agent's next local observation uncertain. This also causes returns to vary across samples that look similar to the actor. DEAL addresses these problems through predictive representation and critic-based reliability weighting.

    Below, Table A shows the component results from Table 2. Table B shows the share of successful runs and their average return over 15 RWARE tasks with 10 seeds each.

    Table Aw/o weightingw/ weightingGain from the weighting
    w/o representation34.2 [31.4, 37.0]41.6 [38.7, 44.3]+7.3 [3.4, 11.2]
    w/ representation47.4 [43.0, 51.9]65.4 [61.3, 69.4]+18.0 [12.0, 24.0]
    Gain from the representation+13.1 [7.8, 18.4]+23.8 [18.9, 28.7]
    Table BReturnSuccessful RunsReturn of the successful runs
    CA-MAPPO backbone34.268%50.3
    Weighting only41.680%51.9
    Predictive only47.465%73.2
    DEAL65.484%77.8

    The predictive representation improves the return of successful runs, while reliability weighting mainly increases how often learning succeeds. Together, they improve both outcomes, producing a gain greater than the sum of their individual improvements. These results show that the two components are not simply separate additions, but reinforce each other within a unified framework.


2. What objective does the weighted PPO update optimize, and when is the original policy optimum preserved? (Q1)

The weighted update optimizes the standard PPO surrogate under a reweighted rollout distribution, and we characterize when the original PPO optimum is preserved and how far it can move otherwise. These results are new in the revised paper (Appendix A.5, summarized in Section 3.2). We briefly describe them below.


  • Optimizing objective.

    Let xx be the critic input, ν(x)\nu(x) its rollout distribution, and w(x)w(x) the positive weight with mean one. DEAL optimizes the PPO surrogate under νw(x)=w(x)ν(x)\nu_w(x)=w(x)\nu(x) (Lemma 4). Since ww is positive and does not depend on the action, min⁡(ρwA^,clip(ρ)wA^)=wmin⁡(ρA^,clip(ρ)A^)\min(\rho w \hat A, \mathrm{clip}(\rho) w \hat A) = w \min(\rho \hat A, \mathrm{clip}(\rho) \hat A). The action distribution and the clipping of every sample are therefore unchanged.


  • Conditions for preserving the optimum.

    We regard the original optimum as the actions that the unweighted PPO update prefers at each local history. These preferences are preserved whenever the weights are uncorrelated with the action-value gaps across the situations that share a local history (Proposition 2). This holds under full observability, and whenever the hidden situation shifts the return of all actions by the same amount.

  • Bounds on deviations.

    The optimum is not preserved in general, and the reason is specific to CTDE. The weights use the critic's centralized input, which the actor does not see. Reweighting situations that the actor can tell apart only changes how fast each is learned, which with shared parameters acts like any other change of the state distribution. A preference can change only when situations that the actor cannot tell apart receive different weights and favor different actions. We bound this effect in three ways.

    1. Within a local history. The change of an action-value gap equals Cov⁡c∣h(w,Qa−Qb)/E[w∣h]\operatorname{Cov}_{c\mid h}(w,Q_a-Q_b)/\mathbb E[w\mid h], where cc is the context hidden from the actor (Proposition 2). The estimated average bound on RWARE is about 12% of a typical gap (Appendix D.4).
    2. Margin condition. A preference whose margin exceeds this bound cannot be reversed.
    3. Bounded suboptimality for any policy class (Theorem 1). Within the trust region, the maximizer of the weighted surrogate loses at most 2ϵs(EΓ^)1/22\epsilon s(\mathbb E\hat \Gamma)^{1/2} on the PPO surrogate, so the loss shrinks in proportion to the spread of the weights. Here ss is the standard deviation of the weights, ϵ\epsilon is the clipping range, and Γ^\hat \Gamma is the decision signal. Since the expected update changes by about 10% of its norm on trained agents (Figure 11b), the weighted update also remains an ascent direction of the PPO surrogate (Proposition 1).

    The weighting rule itself balances these bounds against noise reduction. It minimizes a proxy of the gradient second moment in Proposition 1 plus a penalty on the spread of the weights, and this spread sets all the bounds above (Appendix A.3).

These results concern the surrogate of each update, the level at which PPO itself is usually analyzed. We do not claim a guarantee on the optimal policy of the underlying task. We thank the reviewer for this question, which led us to make the objective of DEAL precise.


3. Does reliability weighting suppress useful learning signals? (W2)

We agree that a large residual can reflect both useful action-value differences and noise. Our new analyses show that DEAL reduces gradient noise while largely preserving useful learning signals. We added empirical gradient diagnostics (Appendix D.4 and Section 4.3) and theoretical analysis (Appendix A.5).

  • Informative learning signals are preserved while gradient noise is reduced.

    On RWARE, samples before team rewards or with the strongest decision signals receive weights similar to other samples. Samples before reward events retain 98% of their gradient contribution on RWARE and 93% on SMAX. The estimated action-dependent part accounts for less than 1% of the advantage variance across all predicted residual levels, while the variability within each action grows with the predicted residual (Figure 11c).

    DEAL reduces policy-gradient variance by 3–10% on RWARE and about 29% early in training on SMAX, while shuffling the same weights increases variance. Our theory also shows that the weighting can change the update only where the weights depart from one and the agent's own action matters (Proposition 1 and Theorem 1). These are the hard samples you raised, and they keep weights similar to other samples. Together, these results suggest that DEAL preserves useful learning signals while reducing gradient noise.

  • The way the weights are computed matters.

    Table C compares different weighting methods while keeping the predictive representation fixed. The new variants were tuned per task, except for the local-information control, which uses the same hyperparameters as the predictive-only variant.

    Table C. WeightsReturn (95% CI)
    No weighting (predictive only)47.4 [43.0, 51.9]
    From the actor's local information (new)46.9 [42.1, 51.7]
    From each sample's realized advantage (new)38.3 [35.1, 41.4]
    From the critic's features, with action-dependent correction (new)57.9 [53.0, 62.7]
    From the critic's features (DEAL)65.4 [61.3, 69.4]

    Removing the action-dependent part of the residual does not improve performance. We introduced a new variant that estimates and removes this component from the residual target. However, it performs below DEAL (Table C). Because the learned correction may introduce noise, we treat this as supporting evidence alongside the diagnostics.

    Weights from the realized advantage are harmful. They depend on the sampled action and can change which actions the policy update favors. They also lower the return below that of no weighting. Weights predicted from the actor's local information provide no improvement either, supporting the importance of centralized information in CTDE. A probe also recovers DEAL's weights from the critic's features but much less from the actor's recurrent state (Appendix D.4).

4. Do the gains come from execution alignment rather than generic auxiliary representation learning? (Q2)

By execution alignment, we mean designing training signals around the information available to each agent during decentralized execution. A generic auxiliary effect would help regardless of what is predicted. Our results show the opposite. The gain depends strongly on the target and is largest for the agent's own local future.

Table D changes only the prediction target, with the same backbone, weighting, and training budget. The first row has no auxiliary prediction at all.

Table D. Prediction targetReturnGain over no prediction
No auxiliary prediction (weighting only)41.6 [38.7, 44.3]
Past joint trajectory20.7 [18.6, 22.8]−20.9
Next global state42.5 [39.0, 45.9]+0.9
Current global state49.0 [45.5, 52.4]+7.5
Own current observation (new)51.6 [47.6, 55.5]+10.0
Own next observation (DEAL)65.4 [61.3, 69.4]+23.8

The three centralized targets are common auxiliary tasks, yet none comes close to DEAL. The new variant reconstructs the agent's current observation instead of predicting its next observation, with all other components unchanged. It gives less than half of DEAL's gain.

  • Centralized targets degrade as the unobserved information grows.

    In the LBF sight sweep (Section 4.3 and Figure 6c), DEAL's improvement grows as the observation range shrinks. To further examine the effect of limited local information, we conducted an analysis of RWARE tasks with more agents on the same map, where a local history determines less of the centralized information. Table E (Table 7 in the revised paper) gives the return of each variant divided by that of DEAL, for all five pairs in RWARE.

    Table E. Prediction targettiny 2 → 4tiny-hard 2 → 4medium 4 → 6large 4 → 8large-hard 4 → 8Average
    Next global state0.93 → 0.820.89 → 0.880.68 → 0.360.71 → 0.200.52 → 0.320.74 → 0.52
    Current global state1.04 → 0.890.83 → 0.840.73 → 0.370.83 → 0.330.99 → 0.470.88 → 0.58
    Past joint trajectory1.05 → 0.070.96 → 0.280.11 → 0.110.51 → 0.080.18 → 0.090.56 → 0.13
    Own current observation0.99 → 0.920.82 → 0.880.81 → 0.720.57 → 0.640.74 → 0.770.79 → 0.79

    On average, all three centralized targets fall when agents are added, while the local target does not change.

These results show that the gain comes from aligning the prediction target with what the agent observes at execution, and not from auxiliary learning in general. If the reviewer meant something different by execution alignment, please let us know, and we will address that reading directly.


Thank you again for your careful review and constructive questions. We believe DEAL indeed provides a useful framework to reduce the decentralized execution gap for widely-considered CTDE. We would be happy to discuss any further questions.

Reviewer_tjR6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DEAL, which improves agent representations by predicting future local observations and adjusts policy update weights according to the critic’s expected residual, thereby improving decentralized multi-agent performance in partially observable environments.

Critical strengths
  1. Adding local future prediction to recurrent MAPPO addresses the actor’s information constraints, while residual weighting accounts for differences in training signals. The two modules target representation learning and optimization, respectively, making the design relatively clear and well-defined.
  2. The paper includes extensive ablation studies covering module activation, alternative prediction targets, a single Gaussian, the number of mixture components, changes in observation range, and hyperparameter sensitivity. These experiments rule out some alternative explanations.
Critical weaknesses

The central issue is that residuals do not represent pure noise. The appendix explicitly states that the residual used during sampling is actually the absolute value of the unnormalized GAE: |G^\lambda - V| = |A|. Although the method predicts the conditional mean of this quantity rather than directly downweighting samples with large advantages, a high expected residual may still indicate situations where action selection truly matters. Consequently, the weighting may also downweight valuable decision contexts. The authors acknowledge this confounding, but the experiments do not disentangle these effects.

Questions for authors

See the Weaknesses.

1 comment

Thank you for the positive evaluation and thoughtful comments. We appreciate your recognition of DEAL's design and extensive ablation studies. We agree that the return residual contains both useful action-value differences and noise, and that its predicted magnitude may also reflect decision-relevant information. To address this concern, we added new gradient diagnostics, an action-dependent correction ablation, and theoretical analysis. We summarize the findings below. Please also see our Common Response for the summary of our responses to all reviewers, and all changes are marked in red{\color{RubineRed}\text{red}} in the revised paper.


1. Does reliability weighting suppress useful learning signals?

Our new analysis suggests that DEAL reduces gradient noise while largely preserving useful learning signals. Importantly, DEAL does not directly weight samples by their realized ∣A∣|A|. Instead, it predicts the expected residual magnitude from the centralized critic's features. The resulting weight depends on the context, not on the sampled action. However, this alone does not guarantee that informative contexts are preserved, so we tested this concern directly.

  • Decision-relevant samples retain their learning signals.

    On RWARE, samples before team rewards and those with the strongest decision signals receive weights similar to other samples. Samples before reward events retain 98% of their gradient contribution. The estimated action-dependent component accounts for less than 1% of the advantage variance across all predicted residual levels, while the variability within each action grows with the predicted residual (Figure 11c). Appendix A.4 now also bounds the action-dependent part of the predicted residual by this decision signal.

  • Gradient noise is reduced without substantially changing the policy update.

    DEAL reduces policy-gradient variance by 3–10% on RWARE and about 29% early in training on SMAX, while shuffling the same weights increases variance. The expected update changes by about 10% of its norm (Figure 11b), close to the estimation noise, but its direction remains nearly unchanged (cosine at least 0.986, Appendix D.4). Our new theoretical analysis also characterizes how weighting can change action preferences (Proposition 2 in Appendix A.5), with an estimated average bound of about 12% of a typical action-value gap on RWARE (Appendix D.4). Our theory also shows that the weighting can change the update only where the weights depart from one and the agent's own action matters (Proposition 1 and Theorem 1). These are the decision contexts you raised, and the measurements above show that they keep weights similar to other samples.

  • Explicitly removing the action-dependent component does not improve performance.

    To test the reviewer's concern more directly, we introduced a new variant that estimates the action-dependent part of the advantage using an additional prediction head and removes it from the residual target. If DEAL's weighting substantially suppressed useful action-value differences, this correction could improve performance. However, the corrected variant performed below DEAL.


Table A. WeightsReturn
No weighting (predictive only)47.4 [43.0, 51.9]
From each sample's realized advantage (new)38.3 [35.1, 41.4]
From the critic's features, with action-dependent correction (new)57.9 [53.0, 62.7]
From the critic's features (DEAL)65.4 [61.3, 69.4]

All variants use the same predictive representation. The new variants were tuned per task. Because the learned correction may introduce additional noise, we treat this result as supporting evidence alongside the gradient diagnostics.

With the predictive representation, reliability weighting reduces the share of collapsed runs from 35% to 16% (Table 8), and the runs that learn reach a higher average return. Thus, weighting improves how often learning succeeds without reducing the performance of successful policies. Together, these results suggest that DEAL reduces gradient noise while largely retaining decision-relevant learning signals.


Thank you again for the thoughtful review and helpful questions. We would be happy to discuss any further questions.

Reviewer_RW2POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In cooperative multi-agent reinforcement learning, a centralized critic can use information from all agents during training, while each actor must make decisions based on its own observation history during execution. This paper introduces DEAL (Decentralized-Execution-Aligned Learning) to reduce this mismatch. The method uses next-observation prediction to improve the actor's representation and adjusts PPO updates based on predicted value residuals from the critic.

Critical strengths

Each component of DEAL has a clear motivation, and both are closely related to the problems discussed.

Instead of reconstructing the global state from limited local observations, the authors predict what each agent may observe at the next time step. This objective better matches the information available during decentralized execution.

DEAL is evaluated on 37 tasks across four benchmarks and achieves the highest mean return on 30 of them. The improvements are especially clear in RWARE, where severe partial observability and sparse rewards make cooperative learning difficult.

Critical weaknesses

The reliability-weighting mechanism needs further justification.

DEAL uses the predicted value residual to measure the reliability of learning signals. Samples with larger predicted residuals receive lower weights during policy updates. However, a large residual does not always mean that a sample is less useful for policy learning. It may reflect environment randomness or meaningful differences in action values. As a result, some informative samples may receive lower weights even when the critic provides reasonably accurate estimates.

Although Appendix A.4 discusses action-dependent residuals, and the method uses predicted average residuals and weight normalization to limit the influence of individual samples, the paper lacks direct experimental evidence on an important question: can the weighting mechanism reduce the influence of unstable learning signals without suppressing useful policy updates?

The contribution of the additional predictive uncertainty feature is unclear.

DEAL uses a GMM to predict the distribution of the next local observation. However, the only additional uncertainty feature the actor receives is the shared log-variance ζ. This feature does not directly include the mixture weights or the differences between component means. It is therefore unclear how well it represents the full predictive uncertainty.

In addition, ζ is computed from the actor's history representation b. The current ablation studies do not separate the benefits of improving the history representation through the auxiliary prediction task from the additional benefits of providing ζ to the actor.

Questions for authors

Q1. DEAL uses the predicted mean absolute residual to measure the reliability of learning signals. However, large residuals may also reflect meaningful differences in action values. Although Appendix A.4 discusses this issue, could the authors further analyze whether the weighting mechanism downweights informative samples during training? How does the method reduce the influence of unstable learning signals while preserving useful policy updates?

Q2. Table 2 shows that reliability weighting improves performance, but I would like to see a comparison with a simpler method. What happens if the weights are based directly on each sample's absolute advantage instead of the predicted average residual? This would help show whether the additional residual prediction network offers any benefit.

Q3. The log-variance feature ζ is computed from the history representation b, which is already provided to the actor. Could the authors add an ablation that removes only ζ from the actor input while keeping the GMM prediction task and loss unchanged?

Q4. Section 3.1 explains why the authors choose ζ, but this only captures GMM variance and does not consider differences between mixtures of components. Could the authors further explain whether ζ provides sufficient uncertainty information for policy decisions? What motivates this choice over other predictive statistics that could also be computed before action selection?

3 comments

Thank you for the positive evaluation and thoughtful comments. We ran the two experiments you asked for and one more variant for Q1, and added new analysis of the weighting. We address the weighting first, then the realized-advantage baseline, and then the role of ζ\zeta. Please also see our Common Response for the summary of our responses to all reviewers, and all changes are marked in red{\color{RubineRed}\text{red}} in the revised paper.


1. Does the weighting downweight informative samples during training? (W1 and Q1)

We agree that a large residual can reflect both useful action-value differences and noise. Our new analysis suggests that DEAL reduces the influence of unstable learning signals while keeping the useful policy updates. We measured this on agents at early, middle, and late stages of training (Appendix D.4 and Section 4.3) and added a theoretical analysis (Appendix A.5). Below, we briefly summarize the key findings and supporting analysis presented in the revised paper.

Table A changes only how the weights are computed. Every row uses the predictive representation, and the new variants are tuned per task. Returns are normalized as in Table 2 of the paper, over 15 RWARE tasks with 10 seeds each.


Table A. WeightsReturn
No weighting (predictive only)47.4 [43.0, 51.9]
From each sample's realized advantage (new)38.3 [35.1, 41.4]
From the critic's features, with action-dependent correction (new)57.9 [53.0, 62.7]
From the critic's features (DEAL)65.4 [61.3, 69.4]

  • The weight depends on the context, not the sampled action.

    DEAL predicts the expected return-residual magnitude from the centralized critic's features and gives smaller weights to contexts with larger predictions. Unlike weighting by realized advantage, the weight does not depend on the sampled action or its realized outcome. Specifically, we use mˉ/(mˉ+m^)\bar m/(\bar m+\hat m), normalized to mean one, where m^\hat m is the predicted residual scale and mˉ\bar m is its batch mean. The weights remain between 0 and 2, and no sample is removed. In practice they stay close to one, with an effective sample size above 98%.


  • Informative learning signals are preserved while gradient noise is reduced.

    On RWARE, samples before team rewards or with the strongest decision signals receive weights similar to other samples. Samples before reward events retain 98% of their gradient contribution on RWARE. The estimated action-dependent part accounts for less than 1% of the advantage variance across all predicted residual levels, while the variability within each action grows with the predicted residual (Figure 11c). We also tested removing the estimated action-dependent component from the residual target, but this did not improve on DEAL (Table A). Because the learned correction may introduce noise, we treat this as supporting evidence alongside the diagnostics.

    DEAL reduces policy-gradient variance by 3–10% on RWARE and about 29% early in training on SMAX, while shuffling the same weights increases variance. Samples with larger predicted residuals contribute more gradient variance and receive smaller weights. Part of this noise comes from teammates' independently sampled actions. It varies even when the critic input is fixed, so a baseline computed from that input cannot remove it, but a weight computed from the same input can give such samples less influence (Appendix A.5). Together, these results suggest that DEAL reduces gradient noise while largely preserving useful learning signals.

    DEAL changes the expected update by about 10% of its norm (Figure 11b), close to the estimation noise, while its direction stays almost unchanged (cosine at least 0.986, Appendix D.4). In theory, the weighted update can be interpreted as the standard PPO surrogate under a reweighted rollout distribution (Lemma 4 in Appendix A.5), and we bound how far an action preference can move (Proposition 2). The theory also shows that the weighting can change the update only where the weights depart from one and the agent's own action matters (Proposition 1 and Theorem 1). These are the informative samples you asked about, and our measurements show that they receive weights similar to other samples. Under full observability, the weighting does not change action preferences. On trained agents, the estimated average bound is about 12% of a typical action-value gap (Appendix D.4). Appendix A.4, which you mentioned, now also bounds the action-dependent part of the residual scale by the decision signal of Theorem 1.

2. What happens if the weights use each sample's absolute advantage? (Q2)

Thank you for suggesting this baseline. Weights from the realized advantage are harmful. Computing the weights from each sample's realized ∣A∣|A| lowers the return below that of the predictive-only variant, without weighting (Table A).

The problem is that these weights depend on the sampled action and can change the relative contributions of different actions to the policy update. They can even reverse action preferences under full observability, which cannot happen with DEAL's action-independent weights (Proposition 2). In our experiments on RWARE, these weights lower gradient variance far more than DEAL, but the expected update changes by about 80% of its norm. Samples before rewards retain only 74% of their gradient contribution on RWARE and 45% on SMAX, compared with 98% and 93% under DEAL.

DEAL avoids this by predicting the residual scale from the centralized context, which explains why its weighting helps while the simpler alternative hurts.


3. What does ζ\zeta add beyond the prediction-trained representation? (W2 and Q3)

Thank you for suggesting this ablation. Because ζ\zeta enters the policy through a stop-gradient, we removed only ζ\zeta from the policy input and kept the mixture prediction head, its loss, and the reliability weighting unchanged. This separates the two benefits, and shows that the prediction-trained representation gives the larger part of the gain, and ζ\zeta adds a further clear gain.

Table B. RepresentationReturn
No auxiliary prediction (weighting only)41.6 [38.7, 44.3]
Mixture prediction loss, without ζ\zeta as policy input (new)57.2 [52.5, 61.8]
Mixture prediction loss with ζ\zeta (DEAL)65.4 [61.3, 69.4]

  • Both components contribute to performance.

    The prediction-trained representation increases the return from 41.6 to 57.2 without ζ\zeta. Providing ζ\zeta raises it further to 65.4, and the 95% bootstrap interval of this difference lies above zero. Although ζ\zeta is computed from bb and adds no new information, it gives the policy direct access to an uncertainty estimate learned by the prediction task. Without ζ\zeta, the policy must learn to extract this quantity from bb through policy learning alone. We therefore view ζ\zeta as a useful inductive bias.


4. Is ζ\zeta sufficient, and why this statistic? (W2 and Q4)

We use ζ\zeta because it provides a simple uncertainty estimate before action selection, without requiring the policy to evaluate the full mixture. We clarified this design choice in Section 3.1.

The shared log-variance ζ\zeta captures uncertainty within mixture components, while the differences between components are modeled through the mixture prediction loss that trains bb. The policy receives both bb and ζ\zeta. Unlike the action-conditioned mixture weights and means, ζ\zeta can be used directly before action selection. Other statistics that account for component differences are possible, but require additional computation and aggregation across actions.

Providing ζ\zeta improves the return (Table B). Replacing the mixture with a single Gaussian also lowers the performance (Table 2), supporting the importance of mixture-based prediction. Even without ζ\zeta, the mixture-trained representation outperforms the single Gaussian and every other prediction target in Table 2, which suggests that bb already carries the between-component information. The mixture-size sweep in Appendix D.1 also shows that small mixtures work best. As discussed in Appendix E, we do not claim that ζ\zeta is a complete summary of the predictive uncertainty. Richer statistics that can be computed before the action, such as the spread between components averaged over actions, are a natural extension.


Thank you again for the thoughtful review and helpful questions. We would be happy to discuss any further questions.

Reviewer_RW2POpenReview ↗

Response

Thanks for the response and new results. I don't have other questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response

We thank all reviewers for their careful reading and constructive reviews. This common reply summarizes the reviews and our responses to the review comments. The details, including the new results, are in our reply to each reviewer. All changes in the paper are marked in red{\color{RubineRed}\text{red}} in the revised PDF.


Strengths

  • Each component has a clear motivation, and the two modules exploit representation learning and policy optimization in a well-defined way (Reviewers RW2P, tjR6).
  • Predicting each agent's own next observation matches the information available at decentralized execution (Reviewer RW2P).
  • The evaluation is broad and statistically solid with extensive ablations and clear gains under severe partial observability (Reviewers RW2P, tjR6, 2m4T).

Raised concerns and our responses

  • Does reliability weighting suppress useful updates? (Reviewers RW2P, tjR6, 2m4T)

    The weights are computed from the predicted residual scale, and the residual equals the absolute advantage. We added a theoretical analysis of the weighted update, gradient measurements on trained agents, and two variants of the weighting. The new results show that DEAL reduces gradient noise while keeping most decision-relevant learning signals.

  • What does the weighted PPO update optimize, and is the original optimum preserved? (Reviewer 2m4T)

    The update optimizes the PPO surrogate under a reweighted rollout distribution. We state the condition under which the original optimum is preserved and bound the change otherwise.

  • What does the uncertainty input ζ\zeta contribute? (Reviewer RW2P)

    We ran the requested ablation that removes only ζ\zeta, which lowers the return, and we explain why ζ\zeta is the statistic given to the policy.

  • Do the gains come from execution alignment, and what is new? (Reviewer 2m4T)

    We clarified how DEAL differs from prior prediction and uncertainty-weighting methods and why its two components comprise a relevant framework to handle the information asymmetry of CTDE. A new variant with a local but non-predictive target supports this.


Revisions of the paper include

  • A theoretical analysis of the weighted PPO objective (Appendix A.5, line 972–1149, summary in Section 3.2, line 279–291).

    • The weighted update is the PPO surrogate under a reweighted rollout distribution (Lemma 4).
    • Weighting lowers gradient variance when smaller weights fall on noisier samples, and its change to the expected update is bounded (Proposition 1).
    • We give the condition under which action preferences are preserved and bound the change otherwise (Proposition 2).
    • For any policy class, we bound how much the maximizer of the weighted surrogate can lose on the PPO surrogate (Theorem 1).
    • We also explain why weights from each sample's realized advantage can reverse action preferences (line 1145).
    • Two standard results are relabeled as Fact 2 and Lemma 3 (Propositions 2 and 3 in the submission).
  • Gradient diagnostics on trained checkpoints (Appendix D.4 with Figure 11, line 1620–1673, summary in Section 4.3, line 473).

    • We measure the quantities of Propositions 1 and 2 on trained RWARE and SMAX agents. DEAL lowers gradient variance while shuffled weights raise it, and the expected update changes little.
    • Samples before rewards and samples with strong decision signals keep most of their influence.
  • Four new variants in Table 2 and Section 4.2 (line 410–414, 453–459, Appendix D.3).

    • Two variants change how the weights are computed. Weights from each sample's realized advantage depend on the sampled action, which the theory excludes. Removing the estimated action-dependent part of the residual would help if the weighting suppressed decision-relevant samples.
    • A variant removes only ζ\zeta from the policy input.
    • A variant reconstructs the agent's current observation, a local but non-predictive target.
  • Effect of team size on the prediction targets (Table 7 in Appendix D.2, line 1563–1576).

    • With more agents on the same map, the centralized targets fall further behind DEAL, while the local current-observation target does not.
  • Clarifications of execution alignment (line 105), the role of ζ\zeta (line 211), and the relation to prior work (line 515).


Experiments during the rebuttal period

  • The four new variants above, evaluated on 15 RWARE tasks with 10 seeds (Table 2).
  • Gradient diagnostics on trained RWARE and SMAX agents (Appendix D.4).
  • A control whose weights are predicted from local information only, reported in our reply to the reviewer 2m4T.

We remain open to further discussion. If any reviewer or the AC has additional questions and comments, we would be glad to further discuss.

Dear Reviewers,

Thank you again for your thoughtful comments and constructive feedback. We have carefully addressed the concerns raised in our rebuttal and revised manuscript, including additional analyses and experiments.

As the discussion period comes to an end, we would greatly appreciate it if you could review our responses and share any further thoughts or questions.

Thank you once again for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.