- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposed Optimal Joint-Action Screening (OJS) to solve the sub-optimality of Value Factorization (VF) methods. OJS applies a adapts the weights of samples for training with a provable threshold, according to their potential interference with optimal-action. Results show that OJS achieves competitive or superior performance over representative value-factorization baselines. The presentation of this paper of this paper is clear, and the method is fairly novel, and the experiments are complete and valid. Therefore, I recommend an accept.
Critical strengths
- Figure 1 provides a clear explanation of the contributions of this work.
- I like the comparison of OJS with other related methods in Appendix B, making the presentation very clear.
- The experiments in this paper are thorough and comprehensive, particularly the experiments on matrix games.
- The experimental section covers plentiful ablation study and sensitivity analysis.
- Though minor, the citation format of this paper is very standard, which has been rare in nowadays conference papers I reviewed.
Critical weaknesses
-
One concern is that the claimed theoretical advantage over heuristic weighting methods is not substantiated. The guarantee holds for any threshold below , so I'm not sure whether such bound is tight and whether it's practical empirically in other domains in general (though it already includes sufficient experiments).
-
My main concern is thatt what is proven in the paper and the value function class actually used in experiments are different. The IGM and theory are based on the state, i.e., state-based IGM [1] and ,in Equation 6--14, but I assume the experimental section actually uses a value function based on observation history (because the authors said this paper is based on CTDE, and normally SMAC use o-a history). It seems this paper is missing a discussion on whether the proposed theory holds true under both state-based and history-based settings.
Questions for authors
-
Doesn't the issue mentioned in this paper (sub-optimality ) generally exist in all VF methods (e.g., the IGM-complete [2] and non-monotonic class QPLEX)? Why is the scope of the study mainly focused on comparing with monotonic-class (e.g, QMIX)? If it is, in the preliminary section, I suggest to merge the VF and IGM, and also discuss the weighted methods like weightedQMIX and QPLEX (qplex appears until the experiment section).
-
Though not sure, but it seems like the proposition 1 and the equality in corollary 1 may not hold as stated. Consider . Its unweighted projection is . Both (A,A) and (B,A) attain , but their true values are 12 and 10. Parts 1 and 2 of Proposition 1 therefore fail; only survives. Can the author explain this case?
-
Without decoupling, the return collapses to 0 even under the mildest penalty (p = −2 in Table 3). What is the explanation for this failure?
References
[1] On Stateful Value Factorization in Multi-Agent Reinforcement Learning, Enrico et. al.
[2] Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning. Baisero et. al.
5 comments
Q2: Correction of Proposition 1 and Corollary 1
We sincerely thank the reviewer for identifying this issue and apologize for the incorrect statements in the original manuscript. We correct the statements as follows:
Proposition 1 (Corrected). Let . Then,
such that .
Corollary 1 (Corrected). Let . Then,
such that . Consequently, .
Specifically, in Proposition 1, we remove part 1 and replace the equality with the inequality in part 2. Importantly, the lower-bound result , which underpins our ideal optimal-action containment guarantee, remains valid.
The corrected proposition is consistent with the reviewer's counterexample.
Why the Original Proposition 1 Fails: Our original derivation of Proposition 1 was based on Propositions 1 and 2 in Appendix B (“Proof of Theorems”) of [2], specialized to the uniform-weighting case. However, Proposition 1 in [2] seems not to extend to this counterexample. In the proof of Proposition 1 in [2], is constructed by replacing with . However, modifying a single joint-action value in this way can violate the monotonicity constraint, so the monotonicity of is not guaranteed, and the proof can not hold for this counterexample.
Proof Sketch of Proposition 1 (Corrected): To prove that such that , assume the contrary: , . We construct by setting for all greedy joint actions, leaving other values unchanged. We choose sufficiently small so that the reduced values remain above both their true values and the next-highest estimated value. This preserves the monotonicity of and reduces the projection error, contradicting . Therefore, Proposition 1 (Corrected) holds. Corollary 1 (Corrected) then follows immediately.
Q3: Failure Without Decoupled Screening
Without decoupling, the primary QMIX both determines the screening threshold and learns from the resulting weights. When , the weight is ; otherwise, . Thus, weighting depends largely on the TD update direction rather than joint-action optimality. Overestimated values can also raise the screening threshold, causing high-return samples to be incorrectly rejected.
We examined a failed training run at .
Table 2. Value estimates and test returns without decoupling at .
| Training steps | Mean Primary Q | Mean Threshold | Mean TD Target | Mean Test Return |
|---|---|---|---|---|
| 0–1M | 15.998 | 16.498 | 14.861 | −10.029 |
| 1–2M | 46.062 | 46.385 | 45.466 | 0.017 |
| 2–3M | 41.664 | 41.756 | 41.402 | 0.000 |
| 3–4M | 26.313 | 26.363 | 26.265 | 0.000 |
| 4–5M | 15.768 | 15.788 | 15.715 | 0.000 |
We define high-return trajectories as those with total episode rewards exceeding 39. Among samples from these trajectories, 85.40% were rejected, and 80.91% were downweighted to approximately . No high-return trajectories appeared in the logged training batches after 1.5M steps. Since had already decreased from 1 to 0.05 within the first 50k steps, successful coordination became difficult to rediscover.
As shown in Table 2, Q-values and screening thresholds initially increased before gradually declining, while test returns remained near zero. This pattern is consistent with early overestimation raising the self-generated threshold, followed by declining bootstrap estimates as successful experiences disappeared. These observations highlight the importance of independent reference-based screening.
References
[1] On Stateful Value Factorization in Multi-Agent Reinforcement Learning, Enrico et al.
[2] Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, Rashid et al.
Dear Reviewer,
Thank you for your constructive feedback. We have uploaded a revised manuscript that incorporates the corrections and additional analyses discussed in our rebuttal. The main revisions addressing your concerns are summarized below:
- Section 2: Expanded the discussion of value factorization to include WQMIX and QPLEX.
- Section 4.1 and Appendix D: Corrected Proposition 1 and Corollary 1 and provided revised proofs.
- Section 5.8: Extended OJS to QPLEX and WQMIX to evaluate its applicability beyond QMIX.
- Appendix E: Extended the theoretical analysis to history-state value functions.
Additionally, we made the following revisions:
- Section 5.7: Added a three-step matrix game to analyze the effectiveness of soft interpolation.
- Appendix F.3: Added experiments to investigate independent bootstrap targets and screening behavior.
- Appendix F.4: Reported training-time and GPU memory overhead.
We appreciate your time and consideration and welcome any further feedback.
We sincerely thank Reviewer DfaX for the encouraging assessment and thoughtful suggestions on the rigor and applicability of our theoretical results.
W1: Threshold Tightness and Practical Applicability
We appreciate the reviewer's insightful observation. However, since the true M(s) is unavailable during learning, a practical threshold must balance tightness and computability. A very loose threshold satisfies the guarantee, but retains too many suboptimal joint actions and therefore provides little screening benefit. The tightest possible threshold is M(s) itself, but M(s)=maxuQ(s,u) depends on the unknown true value function which is unavailable during learning.
OJS instead uses the projected maximum as a theoretically grounded lower-bound threshold, and approximates it with the computable Mref from an independently trained reference QMIX in practice. Moreover, Mref is the maximum value of an independently trained standard QMIX, providing a practical reference threshold that balances tightness and computability. Our experiments demonstrate its effectiveness across diverse benchmarks, and we will clarify this and further investigate its applicability to broader settings. Additionally, the experiments in Q1 (Table 1) show that applying OJS weighting to other VF methods also improves their performance.
W2: Applicability to History-Based Settings
We thank the reviewer for highlighting the gap between theory and practice. The experiments use recurrent individual utilities Qi(τi,ui) and a state-conditioned mixing network, as the reviewer correctly notes. Thus, the practical model is Qtotp(τ,s,u)=fs({Qi(τi,ui)}i∈N), whereas our theoretical analysis considers Qtott(s,u). As discussed in [1], this distinction may affect greedy joint-action selection. However, our projection analysis applies to any fixed conditioning context x, currently instantiated as the global state s, without assumptions on its representation. By augmenting the conditioning context from x=s to x=(s,τ), which additionally incorporates joint action-observation histories, the same proof applies, yielding Mtot(τ,s)≤M(τ,s). This extends the theoretical result to the history-state formulation used in our experiments. We will clarify this distinction and extension in the revised manuscript.
Q1: Applicability Beyond Monotonic VF
We agree that the sub-optimality issue can also arise in broader VF methods, including IGM-complete and non-monotonic methods such as QPLEX. We focus on QMIX because its monotonicity constraint introduces a well-understood representational limitation. We will also revise the Preliminaries to integrate VF and IGM and introduce WQMIX and QPLEX earlier, as suggested.
To further investigate the applicability of OJS, we extend it to QPLEX and WQMIX. The additional results on Predator–Prey are reported below.
Table 1. Mean test returns (five seeds, mean ± 95% CI half-width).
The results show that OJS extends beyond QMIX. OJS-QPLEX substantially improves QPLEX under both penalties, while OJS-WQMIX remains competitive at p=−2 and performs substantially better under p=−5. These results demonstrate the effectiveness of OJS when applied to other VF methods.