- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper makes the case that replay-item utility in off-policy cooperative MARL is mis-estimated when teammates are held frozen: the ranking that maximises a short matched continuation with teammates fixed need not be the ranking that maximises the same continuation once teammates keep learning. The authors name this Co-Adaptive Replay-Value Misalignment (CoRVM) and define three matched branches over a horizon of H learner updates — frz (teammate state fixed), scr (teammate state overwritten at each boundary by a reference-costate path), and co (teammates follow their prescribed updates) — chosen so that reference-scr coincides with reference-co by construction. This yields the three-term identity u^co = u^frz + (u^scr − u^frz) + (u^co − u^scr), termed CAUD, whose middle and last terms are interpreted as baseline complementarity and candidate-induced teammate response (Eq. 3). A two-step Taylor expansion under fixed-target, unclipped plain SGD at H = 2 identifies the leading terms of both components, and a shared-anchor linearisation turns them into a structural score ũ_k(e) = d̄_e^⊤(b_frz + b_comp + b_resp) whose three coefficients are candidate-independent, so each candidate costs one focal gradient plus shared dot products (Eq. 6, Theorem 1). Cross-agent evaluation curvature is avoided by observing that ∇²_RT log p = 0 for non-shared policies on fixed trajectories, leaving a score-function product. A ridge-regularised residual on features φ_k(e) is fitted to sparse paired shadow audits from earlier checkpoints, and the replay batch is drawn without replacement from a softmax over the calibrated scores (Eq. 8, Algorithm C.1).
Validation spans eleven tasks in five cooperative MARL families — SMACv2, LBF, RWARE, Overcooked, GRF — on non-shared QMIX, with four question-specific experiments: an exhaustive batch audit over C(8,2) = 28 candidate batches (RQ1); mechanism audits of the two CAUD channels with Spearman correspondence to first-order predictions (RQ2); a cross-fitted "Oracle Gap Closed" ranking audit over 64 singleton candidates with B_a = 16 (RQ3); and eight-seed matched-resource learning curves reporting nAUC (RQ4). Headline numbers are 68.7% Oracle Gap Closed versus 65.6% for the closest joint-influence control and 45.0% for the strongest replay-priority baseline, and best mean nAUC on 7 of 11 tasks.
Critical strengths
The ∇²_RT log p = 0 observation is a genuinely useful and under-advertised technical contribution. For non-shared decentralised policies with parameter-independent environment laws, the cross-agent log-likelihood Hessian vanishes on fixed trajectories, so the cross-agent curvature term C_RT v_T collapses to the score-function product E[R s_R ⟨s_T, v_T⟩] and requires no cross-agent Hessian–vector product (§3.2.1, Eq. B.16). This is what makes cross-agent influence affordable at all in this setting, it is independent of the CAUD framing, and it generalises to any non-shared CTDE learner with an explicit log-likelihood. It is currently presented as a corollary of the decomposition rather than as a contribution in its own right; stating it as such would survive a revision that reframes the rest of the paper as a cheaper approximation to joint-continuation influence.
Critical weaknesses
The theoretical contribution does not cover the deployed system, and its guarantee is vacuous at the deployed scale. Theorem 1 assumes fixed-target unclipped plain SGD with no optimizer memory at H = 2; deployment uses RMSProp (η = 5×10⁻⁴, decay 0.99), gradient clipping at norm 10, target copies every 200 updates, and H = 4. Proposition 2 bounds suboptimality by 2Bε_k + 2κ_k + Λ_k with B = 32, so ε_k must fall below ΔU_sel/64 for the bound to say anything — and neither ε_k nor κ_k is estimated or bounded, while ΔU_sel is reported only as a normalised percentage and never in return units. Appendix E.3 concedes the diagnostic "is not a uniform certificate of Proposition 2." The fidelity numbers show the gap concretely: the main-text mechanism figure reports ρ = 0.69 and 0.58 at H = 2 where the theorem applies, but at the deployed H = 4 the complementarity and response correlations fall to 0.606 and 0.486 (E.3, Fig. E.2) — a response-channel rank correlation under 0.5 is a weak basis for claiming the mechanism is captured. This is consequential because the paper's framing is "derive the estimator from theory, then deploy it"; as written it derives an estimator for an algorithm it did not run and applies it under a guarantee whose hypothesis is neither checked nor checkable. The one scaling experiment reinforces the concern: at SMACv2 20v20 the Gap advantage over JointInf+Cal is −0.1 and nAUC is −0.8, reported only in Table E.4 and never mentioned in the abstract, introduction, or
Questions for authors
-
What exactly is φ_k(e)? Please give the full feature list, d_φ, and the procedure by which the specification was "fixed before held-out auditing." Given that calibration alone reaches 47.6 Gap — above every published replay-priority baseline — and that the joint effect is superadditive (FrzInf-Only 40.6, Cal-Only 47.6, full 68.7), the reader cannot currently tell whether the calibration term is a residual correction to the structural approximation or a second scorer fitted directly to oracle labels.
-
Please give the values of n_v, α_r, η_0 and η_h, the update-to-environment-step ratio, and the construction of P_k (continuation minibatches, masks, learning-rate and target-copy schedules, random keys). These determine the audit and the score, and D.1 describes itself as listing "hyperparameters" but omits them. Without them the headline tables cannot be rebuilt even with the repository.
-
Either bound ε_k and κ_k empirically — for example, estimate them from the paired probes with a confidence bound and plug them into Proposition 2 alongside ΔU_sel in return units — or downgrade Proposition 2 in the main text from a guarantee to a decomposition of regret into estimable terms, and state plainly that the deployed H = 4 RMSProp system lies outside Theorem 1. Please also move the H = 4 mechanism fidelity (0.606 / 0.486) into the main text as the primary RQ2 evidence.
4 comments
Continuation Protocol and Implementation Details (Reviewer evTx Q2)
Continuation protocol (Q2, continued). All settings are shared across tasks.
- Branches. Each branch clones the complete checkpoint: online and target networks, optimizer states, update counters, schedulers and RNG states.
- Initial update. The candidate replaces one uniformly sampled slot of the reference batch; this update changes only active-owned state (focal agent and mixer).
- Continuation data. One continuation minibatch is sampled from the checkpoint buffer at allocation and reused for all continuation updates; no fresh data are collected inside the window.
- Teammates. co: teammates update normally. frz: all teammate-owned state, including targets, stays fixed. scr: at every boundary, all teammate-owned state is overwritten with the reference-co path. Active-owned state follows its own updates in all three branches.
- Optimizer and targets. Main, audit and shadow learners use RMSProp (, decay 0.99), norm clip 10; shadow counters start from the checkpoint counter. Targets copy every 200 updates for co teammates and each branch's focal agent/mixer. frz skips teammate target copies; scr overwrites teammate online/target/optimizer state with reference-co snapshots at each boundary.
- Random keys. Slot, continuation and evaluation keys are shared across the reference and the three branches; derivative trajectories are disjoint from audit evaluations.
- Cache. On focal access, shared coefficients are refreshed on a cache miss or when their age reaches global learner updates. They are stored with their anchor, protocol and target versions, and are never reused across focal blocks.
- Training clock. The configured rate is one learner update per 100 aggregate joint environment transitions across eight workers; one vector step contributes eight transitions.
Thank you for the thorough and candid rebuttal. The response addresses my requests on scope and disclosure: the deployed learner is now stated to lie outside Theorem 1, Proposition 2 is presented as a conditional bound, the H = 4 fidelity is the primary RQ2 evidence, and the calibration features and continuation protocol are documented. I appreciate this work, and in particular the transparent reporting of inconclusive results.
After careful consideration, I am keeping my rating. My remaining reasons concern what the paper contributes once these corrections are made, rather than points that further clarification could change:
-
The theory–deployment gap is structural. The deployed H = 4 RMSProp/clip/target-copy learner lies outside Theorem 1, and, as the response states, the new diagnostics do not certify B = 32 deployment (positive plug-in margins in 24/66 blocks in the B = 2 exhaustive setting). The theory therefore motivates the score rather than supporting the deployed system. Closing this gap would require new analysis, not rewording.
-
The advantage under matched calibration is inconclusive. In the new common panel, the gain over MAC-PO+Cal is +2.73 with a 95% interval of [−0.21, 5.67] over three seeds. At 20v20 the paired intervals against JointInf+Cal include zero. The remaining case is an accuracy–cost trade-off, which is a narrower claim than the one originally submitted.
-
Response-channel fidelity remains modest (0.486 at the deployed horizon), as the revision acknowledges. This is a property of the estimator rather than of the presentation.
-
Novelty of the identity. I had listed the factorised-policy cross-curvature identity as the main strength. The revision rightly credits it to LOLA; I value the attribution, but it reduces the contribution I had weighed most.
Taken together, these changes substantially alter the paper's theoretical framing and positioning. I believe that version is more accurate and will make a stronger submission, but it deserves a full review of the complete manuscript rather than assessment through a rebuttal thread.
I have no further questions for the authors, and thank them again for their effort.
Clarification of the Contribution and Revision Scope (to Reviewer evTx Q2)
Thank you for the careful follow-up and for acknowledging that the scope and disclosure requests have been addressed. We appreciate your candid assessment and offer three brief clarifications about the contribution and the extent of the revision.
1 Continuity with the submitted work
The revision retains the original CoRVM formulation, matched frozen/scripted/co-adaptive interventions, and ExSpectra estimator and allocation method. The numerical results in Tables 1–2 are unchanged. The revision adds requested controls, proofs and implementation details, and narrows the theoretical and empirical claims. These changes refine the scope and evidence for the original framework.
2 Contribution beyond the score-product identity
We credit the score-product identity to LOLA. Our contribution rests on formalizing replay-ranking reversals under teammate learning, defining the scripted intervention that separates reference-learning effects from candidate-induced response, and using this structure for shared replay scoring. The two reversal constructions demonstrate that either channel alone can change the preferred replay item. Eq. (3) remains exact for the specified matched interventions, including the configured learner; accuracy of the scoring approximation is evaluated separately through audits. We agree that this identity does not extend Theorem 1 to deployment.
3 Evidence beyond rank correlation and the scale trade-off
On the original Table 1 panel, the uncalibrated structural score reaches Gap 59.8 versus 45.0 for the strongest replay-priority baseline. On the common calibration panel, separate channels improve over the ExSpectra and JointInf scalar controls by +3.53 [2.88,4.18] and +3.16 [2.38,3.94], respectively (paired 95% CIs over three seed-level means). These comparisons support structural scoring and channel-wise calibration; superiority over MAC-PO+Cal remains unresolved.
The response channel's correlation is 0.486, as noted. Removing it lowers Gap from 68.7 to 59.7 (Table 1) and lowers the reported mean nAUC on 10/11 tasks (Table 2). This provides additional evidence of decision-relevant signal alongside the component-correlation audit.
Since you noted that the revised version is more accurate, we would be grateful if this strengthened and more carefully scoped version could be reflected in your final evaluation. Thank you again for the detailed review.
Response to Reviewer evTx
We thank the reviewer for the detailed review and for highlighting the usefulness of the factorized-policy identity. Your remedies helped us clarify theoretical scope, test calibration, and detail the protocol. Revisions are marked in blue.
1 Theory and deployment (W, Q3)
§3 states that the deployed H=4 RMSProp/clip/target-copy learner lies outside Theorem 1. Eq. (3) is an exact intervention identity; Theorem 1 gives leading terms for fixed-target, unclipped SGD at H=2; Eq. (6) is a truncated scoring surrogate. RQ2 now reports deployed-learner H=4 fidelity first:
Margins come from a separate family-balanced panel. §4 notes the lower response fidelity. Uncalibrated f+c+r reaches 59.8 versus frozen influence 40.6 and best replay baseline 45.0 on the original panel. Prop. 1 and response-only Prop. 1b show that either channel alone can reverse a decision.
2 Proposition 2 and diagnostics (Q3)
Proposition 2 is now a conditional regret bound with score-error, interaction and sampling terms. At B=32, 64εk+2κk+Λk<ΔUsel,k suffices to certify advantage over the frozen oracle; our audits do not establish it uniformly. A swapped-item refinement (proof in App. B.5) gives: maxSU(S)−E[U(S)]≤E[minc∑e∈S⋆△S∣s(e)−c−u(e)∣]+2κk+Λk. We enumerate all 28 sets in 66 blocks (Ma=8,Ba=2,H=4, 40/80% checkpoints, 200 repeats). J is expected discounted shared-team return (γ=0.99, softmax temperature 0.2, episode limit, no reward rescaling). Column-wise family medians (not additive):
Blockwise, the sampler beats the frozen-utility maximizer under the same estimated labels in 39/66 blocks; median expected advantages range from +0.07 (SMACv2) to +0.54 (LBF). The refinement reduces the bound by a median 25.5%; positive plug-in margins (ΔUsel minus bound) occur in 24/66 versus 10/66 blocks. These finite-pool, estimated-label diagnostics do not certify B=32 deployment.
3 Features and calibration (Q1)
Eight inputs plus an unpenalized intercept: φ=(f,c,r,g); g contains transformed gradient norm, TD magnitude, sequence age, replay count and episode return. g uses historical-window scaling. The specification was fixed during development before held-out auditing. Features, base scores and propensities are stored at generation; fitting/scaling uses only earlier completed probes, with a fixed disjoint source/holdout split.
Residual fitting gives s=(1+βf)f+(1+βc)c+(1+βr)r+βg⊤g+b0, so calibration reweights channels and adds predictions. Cal-Only excludes channels but keeps gradient/TD inputs. Controls share labels, windows and holdouts (this panel differs from Table 1):
Paired t 95% intervals over three seed-level family-balanced means: gains over ExSpectra/JointInf scalars +3.53 [2.88,4.18] / +3.16 [2.38,3.94]; channel-only over no calibration +6.70 [3.68,9.72]; other features +1.23 [-0.26,2.72]; Full over MAC-PO+Cal +2.73 [-0.21,5.67]. Channel reweighting helps. Calibrated replay baselines appear in a separate panel.
4 Reproducibility (Q2)
nv=8 evaluation trajectories per refresh per focal, shared across channels; αr=0.7; one learner update per 100 joint environment transitions. The scoring surrogate uses η0=ηh=0.1 for its displacement, anchor and continuation terms; configured learners use RMSProp at 5×10−4. The full protocol is in the next comment and revised App. C.
5 Positioning and prior work
We approximate joint-continuation influence with shared coefficients and channel calibration. LOLA (Foerster et al., 2017b, App. A.1) derives the cross-policy score-product form; our use removes cross-agent evaluation HVPs, while update-sensitivity HVPs remain.
6 Scaling
At 20v20, ExSpectra is 0.1 Gap and 0.8 nAUC below JointInf+Cal (paired CIs include zero) and +11.5 Gap above w/o CAUD; JointInf takes 2.53× as long per candidate (Table E.4). The abstract, introduction, results and conclusion present this accuracy-cost trade-off.