acceptodds
Under review as a conference paper at ICLR 2027

Optimal Joint-Action Screening for Cooperative Multi-Agent Reinforcement Learning

Abstract

Value factorization enables centralized training with decentralized execution in cooperative multi-agent reinforcement learning, but the widely adopted monotonic mixing structure leads to the suppression of individually optimal actions when they are paired with inferior teammate actions. Existing weighted methods mitigate this interference by identifying and emphasizing potentially optimal joint actions. However, inaccurate optimality criteria may exclude truly optimal samples from the protected set while including non-optimal ones, undermining the intended effect of sample weighting. Instead of identifying optimal joint actions directly, we seek a screening threshold that retains all optimal joint actions while safely identifying non-optimal samples for downweighting. We show that the maximum value from the least-squares projection onto the QMIX function class lower-bounds the optimal joint-action value and therefore provides such a threshold. Based on this, we propose Optimal Joint-Action Screening (OJS), which uses an independently trained reference QMIX to construct the candidate set. Candidate samples receive unit weight, while the remaining samples are adaptively soft-weighted according to their potential interference with optimal-action learning. Experiments across diverse cooperative benchmarks show that OJS achieves competitive or superior performance over representative value-factorization baselines.

Primary area:
reinforcement learning
Keywords:
Cooperative Multi-Agent Reinforcement Learning, Value Factorization, Weighted Value Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DfaXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed Optimal Joint-Action Screening (OJS) to solve the sub-optimality of Value Factorization (VF) methods. OJS applies a adapts the weights of samples for training with a provable threshold, according to their potential interference with optimal-action. Results show that OJS achieves competitive or superior performance over representative value-factorization baselines. The presentation of this paper of this paper is clear, and the method is fairly novel, and the experiments are complete and valid. Therefore, I recommend an accept.

Critical strengths
  1. Figure 1 provides a clear explanation of the contributions of this work.
  2. I like the comparison of OJS with other related methods in Appendix B, making the presentation very clear.
  3. The experiments in this paper are thorough and comprehensive, particularly the experiments on matrix games.
  4. The experimental section covers plentiful ablation study and sensitivity analysis.
  5. Though minor, the citation format of this paper is very standard, which has been rare in nowadays conference papers I reviewed.
Critical weaknesses
  1. One concern is that the claimed theoretical advantage over heuristic weighting methods is not substantiated. The guarantee holds for any threshold below M(s)M(s), so I'm not sure whether such bound is tight and whether it's practical empirically in other domains in general (though it already includes sufficient experiments).

  2. My main concern is thatt what is proven in the paper and the value function class actually used in experiments are different. The IGM and theory are based on the state, i.e., state-based IGM [1] and Q(s,u)Q(s,\mathbf{u}) ,in Equation 6--14, but I assume the experimental section actually uses a value function based on observation history (because the authors said this paper is based on CTDE, and normally SMAC use o-a history). It seems this paper is missing a discussion on whether the proposed theory holds true under both state-based and history-based settings.

Questions for authors
  1. Doesn't the issue mentioned in this paper (sub-optimality ) generally exist in all VF methods (e.g., the IGM-complete [2] and non-monotonic class QPLEX)? Why is the scope of the study mainly focused on comparing with monotonic-class (e.g, QMIX)? If it is, in the preliminary section, I suggest to merge the VF and IGM, and also discuss the weighted methods like weightedQMIX and QPLEX (qplex appears until the experiment section).

  2. Though not sure, but it seems like the proposition 1 and the equality in corollary 1 may not hold as stated. Consider Q=[[12,−100],[10,0]]Q = [[12, −100], [10, 0]]. Its unweighted projection is [[11,−100],[11,0]][[11, −100], [11, 0]]. Both (A,A) and (B,A) attain Mtot=11M_{tot} = 11, but their true values are 12 and 10. Parts 1 and 2 of Proposition 1 therefore fail; only Mtot≤MM_{tot} \leq M survives. Can the author explain this case?

  3. Without decoupling, the return collapses to 0 even under the mildest penalty (p = −2 in Table 3). What is the explanation for this failure?

References

[1] On Stateful Value Factorization in Multi-Agent Reinforcement Learning, Enrico et. al.

[2] Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning. Baisero et. al.

5 comments

We sincerely thank Reviewer DfaX for the encouraging assessment and thoughtful suggestions on the rigor and applicability of our theoretical results.

W1: Threshold Tightness and Practical Applicability

We appreciate the reviewer's insightful observation. However, since the true M(s)M(s) is unavailable during learning, a practical threshold must balance tightness and computability. A very loose threshold satisfies the guarantee, but retains too many suboptimal joint actions and therefore provides little screening benefit. The tightest possible threshold is M(s)M(s) itself, but M(s)=max⁡uQ(s,u)M(s)=\max_{\mathbf{u}}Q(s,\mathbf{u}) depends on the unknown true value function which is unavailable during learning.

OJS instead uses the projected maximum as a theoretically grounded lower-bound threshold, and approximates it with the computable MrefM_{ref} from an independently trained reference QMIX in practice. Moreover, MrefM_{ref} is the maximum value of an independently trained standard QMIX, providing a practical reference threshold that balances tightness and computability. Our experiments demonstrate its effectiveness across diverse benchmarks, and we will clarify this and further investigate its applicability to broader settings. Additionally, the experiments in Q1 (Table 1) show that applying OJS weighting to other VF methods also improves their performance.

W2: Applicability to History-Based Settings

We thank the reviewer for highlighting the gap between theory and practice. The experiments use recurrent individual utilities Qi(τi,ui)Q_i(\tau_i,u_i) and a state-conditioned mixing network, as the reviewer correctly notes. Thus, the practical model is Qtotp(τ,s,u)=fs({Qi(τi,ui)}i∈N)Q_{tot}^p (\boldsymbol{\tau}, s, \mathbf{u}) = f_{s} (\lbrace Q_i(\tau_i,u_i)\rbrace_{i\in N}), whereas our theoretical analysis considers Qtott(s,u)Q_{tot}^t(s,\mathbf{u}). As discussed in [1], this distinction may affect greedy joint-action selection. However, our projection analysis applies to any fixed conditioning context xx, currently instantiated as the global state ss, without assumptions on its representation. By augmenting the conditioning context from x=sx=s to x=(s,τ)x=(s,\boldsymbol{\tau}), which additionally incorporates joint action-observation histories, the same proof applies, yielding Mtot(τ,s)≤M(τ,s)M_{tot}(\boldsymbol{\tau},s)\leq M(\boldsymbol{\tau},s). This extends the theoretical result to the history-state formulation used in our experiments. We will clarify this distinction and extension in the revised manuscript.

Q1: Applicability Beyond Monotonic VF

We agree that the sub-optimality issue can also arise in broader VF methods, including IGM-complete and non-monotonic methods such as QPLEX. We focus on QMIX because its monotonicity constraint introduces a well-understood representational limitation. We will also revise the Preliminaries to integrate VF and IGM and introduce WQMIX and QPLEX earlier, as suggested.

To further investigate the applicability of OJS, we extend it to QPLEX and WQMIX. The additional results on Predator–Prey are reported below.

Table 1. Mean test returns (five seeds, mean ± 95% CI half-width).

Methodp=−2p=-2p=−5p=-5
QMIX0.00 ± 0.000.00 ± 0.00
OJS-QMIX (Ours)39.57 ± 0.5839.89 ± 0.18
QPLEX0.00 ± 0.000.00 ± 0.00
OJS-QPLEX34.32 ± 3.8335.81 ± 1.19
CW-QMIX39.42 ± 0.130.00 ± 0.00
OW-QMIX31.47 ± 21.847.94 ± 22.04
OJS-WQMIX38.97 ± 0.1538.11 ± 0.41

The results show that OJS extends beyond QMIX. OJS-QPLEX substantially improves QPLEX under both penalties, while OJS-WQMIX remains competitive at p=−2p=-2 and performs substantially better under p=−5p=-5. These results demonstrate the effectiveness of OJS when applied to other VF methods.

Q2: Correction of Proposition 1 and Corollary 1

We sincerely thank the reviewer for identifying this issue and apologize for the incorrect statements in the original manuscript. We correct the statements as follows:

Proposition 1 (Corrected). Let Qtot∈ΠQmixQQ_{tot}\in\Pi_{\mathcal{Q}_{mix}}Q. Then,

∀s∈S,∃u^∈arg⁡max⁡uQtot(s,u)\forall s\in S, \exists \hat{\mathbf{u}}\in \arg\max_{\mathbf{u}}Q_{tot}(s,\mathbf{u}) such that Qtot(s,u^)≤Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}}) \le Q(s,\hat{\mathbf{u}}).

Corollary 1 (Corrected). Let Qtot∈ΠQmixQQ_{tot}\in\Pi_{\mathcal{Q}_{mix}}Q. Then,

∀s∈S,∃u^∈arg⁡max⁡uQtot(s,u)\forall s\in S,\exists \hat{\mathbf{u}}\in \arg\max_{\mathbf{u}}Q_{tot}(s,\mathbf{u}) such that Mtot(s)=Qtot(s,u^)≤Q(s,u^)M_{tot}(s)=Q_{tot}(s,\hat{\mathbf{u}})\le Q(s,\hat{\mathbf{u}}). Consequently, Mtot(s)≤M(s)M_{tot}(s)\leq M(s).

Specifically, in Proposition 1, we remove part 1 and replace the equality Qtot(s,u^)=Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}})=Q(s,\hat{\mathbf{u}}) with the inequality Qtot(s,u^)≤Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}})\leq Q(s,\hat{\mathbf{u}}) in part 2. Importantly, the lower-bound result Mtot(s)≤M(s)M_{tot}(s)\leq M(s), which underpins our ideal optimal-action containment guarantee, remains valid.

The corrected proposition is consistent with the reviewer's counterexample.

Why the Original Proposition 1 Fails: Our original derivation of Proposition 1 was based on Propositions 1 and 2 in Appendix B (“Proof of Theorems”) of [2], specialized to the uniform-weighting case. However, Proposition 1 in [2] seems not to extend to this counterexample. In the proof of Proposition 1 in [2], Qtot′Q_{tot}^{\prime} is constructed by replacing Qtot(s,u^)Q_{tot}(s,\hat{\mathbf{u}}) with Q(s,u^)Q(s,\hat{\mathbf{u}}). However, modifying a single joint-action value in this way can violate the monotonicity constraint, so the monotonicity of Qtot′Q_{tot}^\prime is not guaranteed, and the proof can not hold for this counterexample.

Proof Sketch of Proposition 1 (Corrected): To prove that ∃u^∈arg⁡max⁡uQtot(s,u)\exists \hat{\mathbf{u}}\in\arg\max_{\mathbf{u}}Q_{tot}(s,\mathbf{u}) such that Qtot(s,u^)≤Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}})\leq Q(s,\hat{\mathbf{u}}), assume the contrary: ∀u^∈arg⁡max⁡uQtot(s,u)\forall\hat{\mathbf{u}}\in\arg\max_{\mathbf{u}}Q_{tot}(s,\mathbf{u}), Qtot(s,u^)>Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}})>Q(s,\hat{\mathbf{u}}). We construct Qtot′Q_{tot}^{\prime} by setting Qtot′(s,u^)=Qtot(s,u^)−ϵQ_{tot}^{\prime}(s,\hat{\mathbf{u}})=Q_{tot}(s,\hat{\mathbf{u}})-\epsilon for all greedy joint actions, leaving other values unchanged. We choose ϵ>0\epsilon>0 sufficiently small so that the reduced values remain above both their true values Q(s,u^)Q(s,\hat{\mathbf{u}}) and the next-highest estimated value. This preserves the monotonicity of Qtot′Q_{tot}^{\prime} and reduces the projection error, contradicting Qtot∈ΠQmixQQ_{tot}\in\Pi_{\mathcal{Q}_{mix}}Q. Therefore, Proposition 1 (Corrected) holds. Corollary 1 (Corrected) then follows immediately.

Q3: Failure Without Decoupled Screening

Without decoupling, the primary QMIX both determines the screening threshold and learns from the resulting weights. When y<Qtot(s,u)y < Q_{tot}(s,\mathbf{u}), the weight is wmin⁡w_{\min}; otherwise, w>wmin⁡w>w_{\min}. Thus, weighting depends largely on the TD update direction rather than joint-action optimality. Overestimated values can also raise the screening threshold, causing high-return samples to be incorrectly rejected.

We examined a failed training run at p=−2p=-2.

Table 2. Value estimates and test returns without decoupling at p=−2p=-2.

Training stepsMean Primary QMean Threshold MMMean TD Target yyMean Test Return
0–1M15.99816.49814.861−10.029
1–2M46.06246.38545.4660.017
2–3M41.66441.75641.4020.000
3–4M26.31326.36326.2650.000
4–5M15.76815.78815.7150.000

We define high-return trajectories as those with total episode rewards exceeding 39. Among samples from these trajectories, 85.40% were rejected, and 80.91% were downweighted to approximately wmin⁡w_{\min}. No high-return trajectories appeared in the logged training batches after 1.5M steps. Since ϵ\epsilon had already decreased from 1 to 0.05 within the first 50k steps, successful coordination became difficult to rediscover.

As shown in Table 2, Q-values and screening thresholds initially increased before gradually declining, while test returns remained near zero. This pattern is consistent with early overestimation raising the self-generated threshold, followed by declining bootstrap estimates as successful experiences disappeared. These observations highlight the importance of independent reference-based screening.

References

[1] On Stateful Value Factorization in Multi-Agent Reinforcement Learning, Enrico et al.

[2] Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, Rashid et al.

Dear Reviewer,

Thank you for your constructive feedback. We have uploaded a revised manuscript that incorporates the corrections and additional analyses discussed in our rebuttal. The main revisions addressing your concerns are summarized below:

  • Section 2: Expanded the discussion of value factorization to include WQMIX and QPLEX.
  • Section 4.1 and Appendix D: Corrected Proposition 1 and Corollary 1 and provided revised proofs.
  • Section 5.8: Extended OJS to QPLEX and WQMIX to evaluate its applicability beyond QMIX.
  • Appendix E: Extended the theoretical analysis to history-state value functions.

Additionally, we made the following revisions:

  • Section 5.7: Added a three-step matrix game to analyze the effectiveness of soft interpolation.
  • Appendix F.3: Added experiments to investigate independent bootstrap targets and screening behavior.
  • Appendix F.4: Reported training-time and GPU memory overhead.

We appreciate your time and consideration and welcome any further feedback.

Reviewer_tLuVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OJS, which uses an independently trained reference QMIX to identify candidate optimal joint actions and adaptively downweight other samples. The method is motivated by QMIX projection properties and evaluated on matrix games, Predator-Prey, Highway-Env Intersection, SMAC.

Critical strengths
  1. OJS introduces a simple screening rule grounded in QMIX projection properties, providing a principled basis for sample weighting.
  2. Experiments demonstrate substantial gains on coordination tasks and competitive performance across multiple benchmarks.
Critical weaknesses

1.The theoretical novelty is limited. Proposition 1 restates established WQMIX results, and the screening guarantees follow as direct corollaries. 2.OJS trains an additional reference QMIX, but its training-time and memory overhead relative to QMIX and WQMIX is not reported. 3.The main modification to WQMIX is the soft-weighting rule in Eq. (13), which appears heuristic. The limited analysis does not adequately explain the rationale for the interpolation weights.

Questions for authors

What justifies using the target’s relative position between qrefq_{ref} and MrefM_{ref} to determine the interpolation weight in Eq. (13)? How does this quantity reflect interference with optimal-action learning?

5 comments

We thank Reviewer tLuV for evaluating our work and highlighting the need for further analysis. We would also like to clarify the key differences between OJS and WQMIX and the main contributions of our approach.

W1: Theoretical Novelty

While the least-squares projection framework originates from WQMIX [1], OJS derives a new optimal joint-action screening criterion with a containment guarantee, ensuring that all optimal joint actions are retained under ideal projection assumptions.

Moreover, to address approximation errors in practical screening, OJS introduces decoupled screening and soft weighting. Decoupled screening uses an independently trained reference QMIX to prevent self-reinforcing screening errors, while soft weighting preserves learning signals from potentially optimal samples that may be incorrectly rejected. Extensive experiments across multiple cooperative benchmarks demonstrate the effectiveness of OJS, with ablations validating both components.

W2: Training and Memory Overhead

OJS trains an additional reference QMIX network, while OW-QMIX uses a QMIX network and an unrestricted centralized network. We profile all three methods on a single NVIDIA RTX 3090.

Table 1. Estimated full training time (h).

EnvironmentQMIXOW-QMIXOJS
Predator–Prey2.762.882.91
Highway-Env Intersection2.432.782.80
6h_vs_8z8.188.288.55

Table 2. GPU memory usage (MiB).

EnvironmentQMIXOW-QMIXOJS
Predator–Prey143224182350
Highway-Env Intersection584876866
6h_vs_8z102016081538

Training times may vary with episode lengths and evaluation costs. GPU memory reflects observed learner usage.

OJS incurs modest training-time overhead relative to QMIX, with additional GPU memory usage arising from its auxiliary reference network. OJS uses less GPU memory than OW-QMIX across all three environments. The reference network is discarded after training, introducing no additional execution-time overhead.

W3 & Q1: Key Distinction from WQMIX and Rationale for Soft Weighting

Key Distinction from WQMIX:

The main difference between OJS and WQMIX is not the soft-weighting rule, but the theoretically derived optimal joint-action screening criterion. WQMIX employs an unrestricted centralized estimator, with OW-QMIX emphasizing underestimated samples that may be suboptimal and CW-QMIX relying on optimal-action estimates that are sensitive to estimation errors. In contrast, OJS employs an independently trained, monotonic reference QMIX to approximate the projected maximum, based on a theoretically derived screening criterion that guarantees the inclusion of all optimal joint actions under ideal projection assumptions.

Rationale for Soft Weighting:

Soft weighting mitigates practical screening errors caused by noisy TD targets and approximate reference estimates, which may lead to the incorrect rejection of optimal joint-action samples. Rejected samples with targets closer to MrefM_{ref} are more likely to be optimal, whereas those with targets further below MrefM_{ref} are more likely to be suboptimal and potentially interfere with optimal-action learning.

MrefM_{ref} approximates the theoretically derived projected maximum MtotM_{tot}. Since the reference QMIX is trained unweighted, suboptimal joint-action samples may suppress optimal individual utilities, causing qrefq_{ref} to underestimate optimal joint-action values and thus provide a potentially conservative lower endpoint for interpolation.

This motivates the interpolation rule in Eq. (13), and we evaluate its effectiveness in a three-step matrix game with the following payoff matrices:

Step 1:

ABC
A12-20-20
B-201110
C-20109

Step 2:

ABC
A11-2010
B-2012-20
C10-209

Step 3:

ABC
A1110-20
B109-20
C-20-2012

Rows and columns represent the actions of Agents 1 and 2, respectively. Each step's payoff matrix is independent of previous actions. We compare OJS with w/o Soft Weights, which replaces the interpolation rule with hard weighting while retaining the same screening criterion. Both methods use wmin⁡=0.001w_{\min}=0.001 and uniform random exploration.

Table 3. Optimal-policy learning across five seeds (mean ± 95% CI half-width).

MethodStable Optimal Steps ↓Final Optimal SeedsTraining-Averaged Optimal Recognition ↑
OJS64,982 ± 39,2675/50.854 ± 0.089
w/o Soft Weights69,504 ± 27,0005/50.795 ± 0.093

Stable Optimal Steps denotes the earliest training step from which the learned greedy policy remains optimal.

Table 4. False rejection and sample weighting.

MethodP(R∣O)P(R\mid O) (%) ↓Mean ww on O∩R∩w<1O\cap R\cap\\{w < 1\\} ↑Mean ww on II ↓
OJS0.330 ± 0.2460.982 ± 0.0130.00229 ± 0.00139
w/o Soft Weights0.315 ± 0.1980.001 ± 0.0000.00100 ± 0.00000

Here, OO denotes optimal-action samples, RR denotes screening rejection (y<Mrefy < M_{ref}), and II denotes suboptimal joint actions sharing at least one optimal individual action. Metrics are averaged within each seed and then across seeds.

OJS and hard weighting exhibit similar false-rejection rates (0.330% vs. 0.315%). However, among falsely rejected optimal samples that are actually downweighted, OJS assigns substantially higher weights (0.982 vs. 0.001), while maintaining low weights for interfering samples (0.00229). This indicates that the interpolation rule preserves more learning signals from falsely rejected optimal samples and effectively suppresses interference from suboptimal samples. Correspondingly, OJS achieves higher training-averaged optimal-action recognition (0.854 vs. 0.795), providing empirical support for the effectiveness of the interpolation rule in Eq. (13).

References

[1] Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, Rashid et al., NeurIPS 2020.

Dear Reviewer,

Thank you for your constructive feedback. We have uploaded a revised manuscript that incorporates the corrections and additional analyses discussed in our rebuttal. The main revisions addressing your concerns are summarized below:

  • Section 5.7: Added a three-step matrix game to analyze the effectiveness of soft interpolation.
  • Appendix F.4: Reported training-time and GPU memory overhead.

Additionally, we made the following revisions:

  • Section 4.1 and Appendix D: Corrected Proposition 1 and Corollary 1 and provided revised proofs to clarify the theoretical basis of OJS.

  • Section 2: Expanded the discussion of value factorization to include WQMIX and QPLEX.

  • Section 5.8: Extended OJS to QPLEX and WQMIX to evaluate its applicability beyond QMIX.

  • Appendix E: Extended the theoretical analysis to history-state value functions.

  • Appendix F.3: Added experiments to investigate independent bootstrap targets and screening behavior.

We appreciate your time and consideration and welcome any further feedback.

Reviewer_qA3FOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Optimal Joint-Action Screening (OJS), a weighting method for cooperative value factorization. An independently trained, unweighted reference QMIX supplies a screening threshold, while the primary QMIX uses adaptive weights to reduce interference from poorly coordinated joint actions. The method is motivated by projection properties intended to preserve all optimal joint actions in an idealized candidate set. Experiments cover matrix games, Predator-Prey, traffic intersections, SMAC, and SMACv2.

Critical strengths
  • Relevant motivation. The paper addresses a meaningful failure mode of monotonic value factorization: unsuccessful coordination can suppress individual actions that belong to an optimal joint action.
  • Practical design. Separating the reference network from the weighted learner is a sensible design choice. The threshold can be computed through individual greedy actions without enumerating the joint-action space.
  • Supporting experiments. Results are promising in tasks with severe miscoordination penalties, and the component ablations support the practical value of decoupled screening and soft weighting (Section 5 and Appendix E).
Critical weaknesses
  1. Proposition 1 is false as stated, and the exact-matching claim in Corollary 1 consequently fails. Consider the following fixed-state value table:
Agent 1 / Agent 2Action 1Action 2
Action 105
Action 242

Its exact least-squares projection QtotQ_{\mathrm{tot}} onto the function class in Eq. (6) is:

Agent 1 / Agent 2Action 1Action 2
Action 1011/3
Action 211/311/3

The squared error is 14/314/3. Under increasing row and column orderings, the three upper-valued entries are optimally pooled at 11/311/3. Any reversed row or column ordering incurs error at least 88 or 25/225/2, respectively, establishing global optimality. The projected maximizing actions have original values 5,4,25,4,2: none equals 11/311/3, and two are underestimated. This contradicts both parts of Proposition 1, even though the true optimal action is unique. The weaker maximum-value bound, and therefore idealized optimal-action containment, remain valid through a separate clipping argument. However, the current proof and exact-matching statements in Section 4.1 and Appendix D require correction.

  1. The practical implications of the containment result are insufficiently established. Equation (12) replaces the true value with a noisy TD target and the exact projection with a learned reference network. Moreover, Appendix F.3 specifies that the reference constructs its own bootstrap targets. Even with perfect reference values, an optimal action with mean return 22 and terminal targets 11 or 33 with equal probability is downweighted on the y=1y=1 samples when the primary prediction and reference maximum are both 22: the sample is non-candidate and has positive TD error. Thus, optimal-action samples can be downweighted despite an exact reference. The ideal guarantee is explicitly qualified, but its relevance to the implemented screening mechanism needs clearer analysis and targeted validation.

  2. The weighting definition is incomplete. Equation (13) omits the δ≤0\delta\le0 unit-weight condition stated in Section 4.3 and Algorithm 1. The complete rule should be specified consistently. This is a formulation issue; the manuscript alone does not establish whether the implementation is affected.

The incorrect projection statements and the unresolved connection between ideal containment and practical screening weaken the paper's central theoretical motivation. I would reconsider my recommendation if the rebuttal corrects the proof and clarifies the scope and practical support of the resulting claims.

Questions for authors
  1. Can you address the counterexample above and provide corrected statements and proofs for Proposition 1 and Corollary 1? Please identify which subsequent claims remain valid and which require revision.
  2. What supports comparing the primary TD target against a reference maximum learned from different bootstrap targets? Please clarify the guarantee's scope and provide an error analysis or a controlled experiment with known optimal actions, measuring how often their samples are excluded or downweighted.
  3. Does the implementation assign unit weight whenever δ≤0\delta\le0? Please reconcile Eq. (13) with Algorithm 1 and state the complete weighting rule.
5 comments

We sincerely appreciate Reviewer qA3F for the careful theoretical analysis and insightful suggestions on practical screening.

W1 & Q1. Correction of Proposition 1 and Corollary 1

We correct Proposition 1 and Corollary 1 as follows:

Proposition 1 (Corrected). Let Qtot∈ΠQmixQQ_{tot}\in\Pi_{\mathcal{Q}_{mix}}Q. Then,

∀s∈S,∃u^∈arg⁡max⁡uQtot(s,u)\forall s\in S, \exists \hat{\mathbf{u}}\in \arg\max_{\mathbf{u}}Q_{tot}(s,\mathbf{u}) such that Qtot(s,u^)≤Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}}) \le Q(s,\hat{\mathbf{u}}).

Corollary 1 (Corrected). Let Qtot∈ΠQmixQQ_{tot}\in\Pi_{\mathcal{Q}_{mix}}Q. Then,

∀s∈S,∃u^∈arg⁡max⁡uQtot(s,u)\forall s\in S,\exists \hat{\mathbf{u}}\in \arg\max_{\mathbf{u}}Q_{tot}(s,\mathbf{u}) such that Mtot(s)=Qtot(s,u^)≤Q(s,u^)M_{tot}(s)=Q_{tot}(s,\hat{\mathbf{u}})\le Q(s,\hat{\mathbf{u}}). Consequently, Mtot(s)≤M(s)M_{tot}(s)\leq M(s).

Specifically, in Proposition 1, we remove part 1 and replace the equality Qtot(s,u^)=Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}})=Q(s,\hat{\mathbf{u}}) with the inequality Qtot(s,u^)≤Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}})\leq Q(s,\hat{\mathbf{u}}) in part 2. The conclusion Mtot(s)≤M(s)M_{tot}(s)\leq M(s) can still be derived, so this correction does not alter the screening rule or the OJS algorithm. The counterexample is consistent with Proposition 1 (Corrected), since two of the projected-greedy joint actions are underestimated.

Why the Original Proposition 1 Fails: Our original derivation of Proposition 1 was based on Propositions 1 and 2 in Appendix B (“Proof of Theorems”) of [1], specialized to the uniform-weighting case. However, Proposition 1 in [1] seems not to extend to this counterexample. In the proof of Proposition 1 in [1], Qtot′Q_{tot}^{\prime} is constructed by replacing Qtot(s,u^)Q_{tot}(s,\hat{\mathbf{u}}) with Q(s,u^)Q(s,\hat{\mathbf{u}}). However, modifying a single joint-action value in this way can violate the monotonicity constraint, so the monotonicity of Qtot′Q_{tot}^\prime is not guaranteed, and the proof can not hold for this counterexample.

Proof Sketch of Proposition 1 (Corrected): To prove that ∃u^∈arg⁡max⁡uQtot(s,u)\exists \hat{\mathbf{u}}\in\arg\max_{\mathbf{u}}Q_{tot}(s,\mathbf{u}) such that Qtot(s,u^)≤Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}})\leq Q(s,\hat{\mathbf{u}}), assume the contrary: ∀u^∈arg⁡max⁡uQtot(s,u)\forall\hat{\mathbf{u}}\in\arg\max_{\mathbf{u}}Q_{tot}(s,\mathbf{u}), Qtot(s,u^)>Q(s,u^)Q_{tot}(s,\hat{\mathbf{u}})>Q(s,\hat{\mathbf{u}}). We construct Qtot′Q_{tot}^{\prime} by setting Qtot′(s,u^)=Qtot(s,u^)−ϵQ_{tot}^{\prime}(s,\hat{\mathbf{u}})=Q_{tot}(s,\hat{\mathbf{u}})-\epsilon for all greedy joint actions, leaving other values unchanged. We choose ϵ>0\epsilon>0 sufficiently small so that the reduced values remain above both their true values Q(s,u^)Q(s,\hat{\mathbf{u}}) and the next-highest estimated value. This preserves the monotonicity of Qtot′Q_{tot}^{\prime} and reduces the projection error, contradicting Qtot∈ΠQmixQQ_{tot}\in\Pi_{\mathcal{Q}_{mix}}Q. Therefore, Proposition 1 (Corrected) holds. Corollary 1 (Corrected) then follows immediately.

Corollary 2 and the optimal joint-action containment guarantee remain valid. The OJS algorithm and Eqs. (6)–(14) remain unchanged, except for the Eq. (13) correction discussed in W3. We will revise the corresponding descriptions in Section 4.1, Appendix D, and Section 5.1 accordingly.

W2: Practical Implications of the Containment Result

We acknowledge that noisy TD targets and an approximate reference may cause optimal-action samples to be incorrectly rejected. Our soft-weighting mechanism (Section 4.3) mitigates this issue by retaining learning signals from potentially optimal samples.

To validate this, we construct a three-step matrix game with the following payoff matrices:

Step 1:

ABC
A12-20-20
B-201110
C-20109

Step 2:

ABC
A11-2010
B-2012-20
C10-209

Step 3:

ABC
A1110-20
B109-20
C-20-2012

Rows and columns represent the actions of Agents 1 and 2, respectively. Each step's payoff matrix is independent of previous actions. We compare OJS with its hard-weighting variant (w/o Soft Weights), using wmin⁡=0.001w_{\min}=0.001 and uniform random exploration.

Table 1. Optimal-policy learning across five seeds (mean ± 95% CI half-width).

MethodStable Optimal Steps ↓Final Optimal SeedsTraining-Averaged Optimal Recognition ↑
OJS64,982 ± 39,2675/50.854 ± 0.089
w/o Soft Weights69,504 ± 27,0005/50.795 ± 0.093

Stable Optimal Steps denotes the earliest training step from which the learned greedy policy remains optimal.

Table 2. False rejection and sample weighting.

MethodP(R∣O)P(R\mid O) (%) ↓Mean ww on O∩R∩w<1O\cap R \cap \\{w < 1\\} ↑Mean ww on II ↓
OJS0.330 ± 0.2460.982 ± 0.0130.00229 ± 0.00139
w/o Soft Weights0.315 ± 0.1980.001 ± 0.0000.00100 ± 0.00000

Here, OO denotes optimal-action samples, RR denotes screening rejection (y<Mrefy < M_{ref}), and II denotes suboptimal joint actions sharing at least one optimal individual action. Metrics are averaged within each seed and then across seeds.

Analysis. False-rejection rates are comparable (0.330% vs. 0.315%). Among falsely rejected optimal samples, those actually downweighted (w<1w < 1) receive much higher mean weights under OJS (0.982 vs. 0.001). Meanwhile, OJS maintains low weights for interfering samples (0.00229) and achieves higher training-averaged optimal recognition (0.854 vs. 0.795). These results show that soft weighting preserves learning signals from falsely rejected optimal samples while suppressing interference, supporting its practical effectiveness despite imperfect screening.

W3 & Q3: The weighting definition is incomplete.

The implementation assigns unit weight whenever δ≤0\delta\leq0, as specified in Section 4.3 and Algorithm 1. We acknowledge that Eq. (13) omits the δ≤0\delta\leq0 unit-weight condition. We will correct Eq. (13) as follows:

\begin{cases} 1, & y\geq M_{ref}(s)\ \text{or}\ \delta\leq0,\\\\ w_{\min}, & y<q_{ref}(s,\mathbf{u}),\\\\ w_{\min}+(1-w_{\min})\dfrac{y-q_{ref}(s,\mathbf{u})}{M_{ref}(s)-q_{ref}(s,\mathbf{u})}, & \text{otherwise}.\end{cases}$$ where $\delta=Q_{tot}(s,\mathbf{u})-y$. # Q2: Practical Screening with Different Bootstrap Targets Both the primary TD target and the reference maximum are estimated using the same training data, reward function, and discount factor, and are therefore on the same value scale. We evaluate two controls in the three-step matrix game: **Shared TD Targets** trains the separate reference network using the primary TD targets, while **w/o Decoupling** uses the primary QMIX itself for screening and weighting. **Table 3. Optimal-policy learning.** |Method|Stable Optimal Steps ↓|Final Optimal Seeds|Training-Averaged Optimal Recognition ↑| |:---:|:---:|:---:|:---:| |OJS|**64,982 ± 39,267**|5/5|**0.854 ± 0.089**| |Shared TD Targets|71,707 ± 36,041|5/5|0.820 ± 0.137| |w/o Decoupling|91,051 ± 10,710|5/5|0.794 ± 0.101| Table 4 further examines the Step 1 screening behavior. **Table 4. Step 1 screening statistics.** |Method|$y^P-M_{screen}$|$P(R\mid O)$ (%) ↓|Mean $w$ on $O\cap R$ ↑| |:---:|:---:|:---:|:---:| |OJS|**3.177 ± 0.130**|**0.154 ± 0.089**|0.997 ± 0.006| |Shared TD Targets|1.241 ± 0.158|6.583 ± 4.316|**0.999 ± 0.001**| |w/o Decoupling|0.134 ± 0.127|35.707 ± 19.077|0.324 ± 0.350| Here, $y^P$ is the primary TD target for sampled optimal actions, $M_{screen}$ is the screening maximum, and $P(R\mid O)$ measures the proportion of sampled optimal actions rejected by $y^P < M_{screen}$. **Analysis.** OJS achieves a larger screening margin than Shared TD Targets (3.177 vs. 1.241), reducing false rejection from 6.583% to 0.154%. Without decoupling, false rejection increases to 35.707%, and the mean weight of rejected optimal samples drops to 0.324. These results suggest that **independent reference bootstrapping provides a more stable screening threshold that better preserves optimal samples**. **References** [1] Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, Rashid et al., NeurIPS 2020.

Dear Reviewer,

Thank you for your constructive feedback. We have uploaded a revised manuscript that incorporates the corrections and additional analyses discussed in our rebuttal. The main revisions addressing your concerns are summarized below:

  • Section 4.1 and Appendix D: Corrected Proposition 1 and Corollary 1 and provided revised proofs.
  • Section 5.7: Added a three-step matrix game to analyze the effectiveness of soft interpolation under imperfect screening.
  • Appendix F.3: Added experiments to investigate independent bootstrap targets and screening behavior.
  • Section 4.3: Corrected Eq. (13) to explicitly include the unit-weight condition for δ≤0\delta\leq0.

Additionally, we made the following revisions:

  • Section 2: Expanded the discussion of value factorization to include WQMIX and QPLEX.
  • Section 5.8: Extended OJS to QPLEX and WQMIX to evaluate its applicability beyond QMIX.
  • Appendix E: Extended the theoretical analysis to history-state value functions.
  • Appendix F.4: Reported training-time and GPU memory overhead.

We appreciate your time and consideration and welcome any further feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.