acceptodds
Under review as a conference paper at ICLR 2027

What Teammates Learn Next Changes Which Updates Are Worth Making

Abstract

Off-policy cooperative multi-agent reinforcement learning (MARL) reuses past interactions through experience replay, but limited update budgets permit learning from only a fraction of candidates, while each update's team utility depends on teammates' subsequent learning and responses. However, existing replay and data-attribution criteria largely use current policies and error signals, without accounting for subsequent teammate learning. The most valuable update with teammates held fixed need not remain optimal under co-adaptation, a phenomenon we call Co-Adaptive Replay-Value Misalignment (CoRVM). To address CoRVM, we propose ExSpectra, which estimates co-adaptive update utility before replay allocation. Co-Adaptive Utility Decomposition (CAUD) exactly decomposes co-adaptation-induced utility changes under fixed-data matched interventions into baseline teammate-learning and update-induced teammate-response effects. Based on CAUD, we construct a first-order local influence estimator that augments frozen-teammate utility with co-adaptive corrections using cross-agent evaluation curvature and update sensitivity. Shared Hessian–vector products and cross-agent curvature estimates amortize second-order computation across candidates, while sparse paired probes calibrate residual error. Held-out matched audits across five benchmark families confirm that CoRVM changes finite-budget replay decisions. Extensive experiments show that ExSpectra outperforms strong replay-based and resource-matched baselines on most tasks. Code is anonymously available at https://anonymous.4open.science/r/ExSpectra-0DC0/

Primary area:
reinforcement learning
Keywords:
Cooperative Multi-Agent Reinforcement Learning, Experience Replay, Co-Adaptation, Replay-Value Misalignment, Data Attribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_evTxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper makes the case that replay-item utility in off-policy cooperative MARL is mis-estimated when teammates are held frozen: the ranking that maximises a short matched continuation with teammates fixed need not be the ranking that maximises the same continuation once teammates keep learning. The authors name this Co-Adaptive Replay-Value Misalignment (CoRVM) and define three matched branches over a horizon of H learner updates — frz (teammate state fixed), scr (teammate state overwritten at each boundary by a reference-costate path), and co (teammates follow their prescribed updates) — chosen so that reference-scr coincides with reference-co by construction. This yields the three-term identity u^co = u^frz + (u^scr − u^frz) + (u^co − u^scr), termed CAUD, whose middle and last terms are interpreted as baseline complementarity and candidate-induced teammate response (Eq. 3). A two-step Taylor expansion under fixed-target, unclipped plain SGD at H = 2 identifies the leading terms of both components, and a shared-anchor linearisation turns them into a structural score ũ_k(e) = d̄_e^⊤(b_frz + b_comp + b_resp) whose three coefficients are candidate-independent, so each candidate costs one focal gradient plus shared dot products (Eq. 6, Theorem 1). Cross-agent evaluation curvature is avoided by observing that ∇²_RT log p = 0 for non-shared policies on fixed trajectories, leaving a score-function product. A ridge-regularised residual on features φ_k(e) is fitted to sparse paired shadow audits from earlier checkpoints, and the replay batch is drawn without replacement from a softmax over the calibrated scores (Eq. 8, Algorithm C.1).

Validation spans eleven tasks in five cooperative MARL families — SMACv2, LBF, RWARE, Overcooked, GRF — on non-shared QMIX, with four question-specific experiments: an exhaustive batch audit over C(8,2) = 28 candidate batches (RQ1); mechanism audits of the two CAUD channels with Spearman correspondence to first-order predictions (RQ2); a cross-fitted "Oracle Gap Closed" ranking audit over 64 singleton candidates with B_a = 16 (RQ3); and eight-seed matched-resource learning curves reporting nAUC (RQ4). Headline numbers are 68.7% Oracle Gap Closed versus 65.6% for the closest joint-influence control and 45.0% for the strongest replay-priority baseline, and best mean nAUC on 7 of 11 tasks.

Critical strengths

The ∇²_RT log p = 0 observation is a genuinely useful and under-advertised technical contribution. For non-shared decentralised policies with parameter-independent environment laws, the cross-agent log-likelihood Hessian vanishes on fixed trajectories, so the cross-agent curvature term C_RT v_T collapses to the score-function product E[R s_R ⟨s_T, v_T⟩] and requires no cross-agent Hessian–vector product (§3.2.1, Eq. B.16). This is what makes cross-agent influence affordable at all in this setting, it is independent of the CAUD framing, and it generalises to any non-shared CTDE learner with an explicit log-likelihood. It is currently presented as a corollary of the decomposition rather than as a contribution in its own right; stating it as such would survive a revision that reframes the rest of the paper as a cheaper approximation to joint-continuation influence.

Critical weaknesses

The theoretical contribution does not cover the deployed system, and its guarantee is vacuous at the deployed scale. Theorem 1 assumes fixed-target unclipped plain SGD with no optimizer memory at H = 2; deployment uses RMSProp (η = 5×10⁻⁴, decay 0.99), gradient clipping at norm 10, target copies every 200 updates, and H = 4. Proposition 2 bounds suboptimality by 2Bε_k + 2κ_k + Λ_k with B = 32, so ε_k must fall below ΔU_sel/64 for the bound to say anything — and neither ε_k nor κ_k is estimated or bounded, while ΔU_sel is reported only as a normalised percentage and never in return units. Appendix E.3 concedes the diagnostic "is not a uniform certificate of Proposition 2." The fidelity numbers show the gap concretely: the main-text mechanism figure reports ρ = 0.69 and 0.58 at H = 2 where the theorem applies, but at the deployed H = 4 the complementarity and response correlations fall to 0.606 and 0.486 (E.3, Fig. E.2) — a response-channel rank correlation under 0.5 is a weak basis for claiming the mechanism is captured. This is consequential because the paper's framing is "derive the estimator from theory, then deploy it"; as written it derives an estimator for an algorithm it did not run and applies it under a guarantee whose hypothesis is neither checked nor checkable. The one scaling experiment reinforces the concern: at SMACv2 20v20 the Gap advantage over JointInf+Cal is −0.1 and nAUC is −0.8, reported only in Table E.4 and never mentioned in the abstract, introduction, or

Questions for authors
  1. What exactly is φ_k(e)? Please give the full feature list, d_φ, and the procedure by which the specification was "fixed before held-out auditing." Given that calibration alone reaches 47.6 Gap — above every published replay-priority baseline — and that the joint effect is superadditive (FrzInf-Only 40.6, Cal-Only 47.6, full 68.7), the reader cannot currently tell whether the calibration term is a residual correction to the structural approximation or a second scorer fitted directly to oracle labels.

  2. Please give the values of n_v, α_r, η_0 and η_h, the update-to-environment-step ratio, and the construction of P_k (continuation minibatches, masks, learning-rate and target-copy schedules, random keys). These determine the audit and the score, and D.1 describes itself as listing "hyperparameters" but omits them. Without them the headline tables cannot be rebuilt even with the repository.

  3. Either bound ε_k and κ_k empirically — for example, estimate them from the paired probes with a confidence bound and plug them into Proposition 2 alongside ΔU_sel in return units — or downgrade Proposition 2 in the main text from a guarantee to a decomposition of regret into estimable terms, and state plainly that the deployed H = 4 RMSProp system lies outside Theorem 1. Please also move the H = 4 mechanism fidelity (0.606 / 0.486) into the main text as the primary RQ2 evidence.

4 comments
AuthorsOpenReview ↗

Response to Reviewer evTx

We thank the reviewer for the detailed review and for highlighting the usefulness of the factorized-policy identity. Your remedies helped us clarify theoretical scope, test calibration, and detail the protocol. Revisions are marked in blue.

1 Theory and deployment (W, Q3)

§3 states that the deployed H=4H=4 RMSProp/clip/target-copy learner lies outside Theorem 1. Eq. (3) is an exact intervention identity; Theorem 1 gives leading terms for fixed-target, unclipped SGD at H=2H=2; Eq. (6) is a truncated scoring surrogate. RQ2 now reports deployed-learner H=4H=4 fidelity first:

EvidenceComplementarityResponse
Original rank correlation0.6060.486
New same-target margin over other predictor+0.281+0.141
Gap after removing channel (full: 68.7)61.959.7

Margins come from a separate family-balanced panel. §4 notes the lower response fidelity. Uncalibrated f+c+rf+c+r reaches 59.8 versus frozen influence 40.6 and best replay baseline 45.0 on the original panel. Prop. 1 and response-only Prop. 1b show that either channel alone can reverse a decision.

2 Proposition 2 and diagnostics (Q3)

Proposition 2 is now a conditional regret bound with score-error, interaction and sampling terms. At B=32B=32, 64εk+2κk+Λk<ΔUsel,k64\varepsilon_k+2\kappa_k+\Lambda_k<\Delta U_{\rm sel,k} suffices to certify advantage over the frozen oracle; our audits do not establish it uniformly. A swapped-item refinement (proof in App. B.5) gives: max⁡SU(S)−E[U(S^)]≤E[min⁡c ⁣ ⁣∑e∈S⋆△S^ ⁣∣s(e)−c−u(e)∣]+2κk+Λk.\max_S U(S)-\mathbb E[U(\widehat S)]\le\mathbb E\Big[\min_c\!\!\sum_{e\in S^\star\triangle\widehat S}\!|s(e)-c-u(e)|\Big]+2\kappa_k+\Lambda_k. We enumerate all 28 sets in 66 blocks (Ma=8,Ba=2,H=4M_a=8,B_a=2,H=4, 40/80% checkpoints, 200 repeats). JJ is expected discounted shared-team return (γ=0.99\gamma=0.99, softmax temperature 0.2, episode limit, no reward rescaling). Column-wise family medians (not additive):

FamilyΔUsel\Delta U_{\rm sel}ε^\hat\varepsilonκ^\hat\kappaΛ\LambdaEq. (9)RefinedExpected regret
SMACv2.735.146.195.7181.6611.189.698
LBF1.096.111.198.7141.5841.155.692
RWARE.850.153.192.7041.7261.212.699
Overcooked1.096.160.208.7121.7361.245.767
GRF.995.121.193.6951.5391.189.658

Blockwise, the sampler beats the frozen-utility maximizer under the same estimated labels in 39/66 blocks; median expected advantages range from +0.07 (SMACv2) to +0.54 (LBF). The refinement reduces the bound by a median 25.5%; positive plug-in margins (ΔUsel\Delta U_{\rm sel} minus bound) occur in 24/66 versus 10/66 blocks. These finite-pool, estimated-label diagnostics do not certify B=32B=32 deployment.

3 Features and calibration (Q1)

Eight inputs plus an unpenalized intercept: φ=(f,c,r,g)\varphi=(f,c,r,g); gg contains transformed gradient norm, TD magnitude, sequence age, replay count and episode return. gg uses historical-window scaling. The specification was fixed during development before held-out auditing. Features, base scores and propensities are stored at generation; fitting/scaling uses only earlier completed probes, with a fixed disjoint source/holdout split.

Residual fitting gives s=(1+βf)f+(1+βc)c+(1+βr)r+βg⊤g+b0s=(1+\beta_f)f+(1+\beta_c)c+(1+\beta_r)r+\beta_g^\top g+b_0, so calibration reweights channels and adds predictions. Cal-Only excludes channels but keeps gradient/TD inputs. Controls share labels, windows and holdouts (this panel differs from Table 1):

ModelGap
Three channels + other features68.23
ExSpectra / JointInf scalar + other features64.70 / 65.07
Channel-only calibration67.00
Cal-Only / no calibration48.20 / 60.30
MAC-PO / DIFFER score + other features65.50 / 64.30

Paired t 95% intervals over three seed-level family-balanced means: gains over ExSpectra/JointInf scalars +3.53 [2.88,4.18] / +3.16 [2.38,3.94]; channel-only over no calibration +6.70 [3.68,9.72]; other features +1.23 [-0.26,2.72]; Full over MAC-PO+Cal +2.73 [-0.21,5.67]. Channel reweighting helps. Calibrated replay baselines appear in a separate panel.

4 Reproducibility (Q2)

nv=8n_v=8 evaluation trajectories per refresh per focal, shared across channels; αr=0.7\alpha_r=0.7; one learner update per 100 joint environment transitions. The scoring surrogate uses η0=ηh=0.1\eta_0=\eta_h=0.1 for its displacement, anchor and continuation terms; configured learners use RMSProp at 5×10−45\times10^{-4}. The full protocol is in the next comment and revised App. C.

5 Positioning and prior work

We approximate joint-continuation influence with shared coefficients and channel calibration. LOLA (Foerster et al., 2017b, App. A.1) derives the cross-policy score-product form; our use removes cross-agent evaluation HVPs, while update-sensitivity HVPs remain.

6 Scaling

At 20v20, ExSpectra is 0.1 Gap and 0.8 nAUC below JointInf+Cal (paired CIs include zero) and +11.5 Gap above w/o CAUD; JointInf takes 2.53× as long per candidate (Table E.4). The abstract, introduction, results and conclusion present this accuracy-cost trade-off.

AuthorsOpenReview ↗

Continuation Protocol and Implementation Details (Reviewer evTx Q2)

Continuation protocol (Q2, continued). All settings are shared across tasks.

  • Branches. Each branch clones the complete checkpoint: online and target networks, optimizer states, update counters, schedulers and RNG states.
  • Initial update. The candidate replaces one uniformly sampled slot of the reference batch; this update changes only active-owned state (focal agent and mixer).
  • Continuation data. One continuation minibatch is sampled from the checkpoint buffer at allocation and reused for all H−1=3H-1=3 continuation updates; no fresh data are collected inside the window.
  • Teammates. co: teammates update normally. frz: all teammate-owned state, including targets, stays fixed. scr: at every boundary, all teammate-owned state is overwritten with the reference-co path. Active-owned state follows its own updates in all three branches.
  • Optimizer and targets. Main, audit and shadow learners use RMSProp (5×10−45\times10^{-4}, decay 0.99), norm clip 10; shadow counters start from the checkpoint counter. Targets copy every 200 updates for co teammates and each branch's focal agent/mixer. frz skips teammate target copies; scr overwrites teammate online/target/optimizer state with reference-co snapshots at each boundary.
  • Random keys. Slot, continuation and evaluation keys are shared across the reference and the three branches; derivative trajectories are disjoint from audit evaluations.
  • Cache. On focal access, shared coefficients are refreshed on a cache miss or when their age reaches Kv=200K_v=200 global learner updates. They are stored with their anchor, protocol and target versions, and are never reused across focal blocks.
  • Training clock. The configured rate is one learner update per 100 aggregate joint environment transitions across eight workers; one vector step contributes eight transitions.

Thank you for the thorough and candid rebuttal. The response addresses my requests on scope and disclosure: the deployed learner is now stated to lie outside Theorem 1, Proposition 2 is presented as a conditional bound, the H = 4 fidelity is the primary RQ2 evidence, and the calibration features and continuation protocol are documented. I appreciate this work, and in particular the transparent reporting of inconclusive results.

After careful consideration, I am keeping my rating. My remaining reasons concern what the paper contributes once these corrections are made, rather than points that further clarification could change:

  1. The theory–deployment gap is structural. The deployed H = 4 RMSProp/clip/target-copy learner lies outside Theorem 1, and, as the response states, the new diagnostics do not certify B = 32 deployment (positive plug-in margins in 24/66 blocks in the B = 2 exhaustive setting). The theory therefore motivates the score rather than supporting the deployed system. Closing this gap would require new analysis, not rewording.

  2. The advantage under matched calibration is inconclusive. In the new common panel, the gain over MAC-PO+Cal is +2.73 with a 95% interval of [−0.21, 5.67] over three seeds. At 20v20 the paired intervals against JointInf+Cal include zero. The remaining case is an accuracy–cost trade-off, which is a narrower claim than the one originally submitted.

  3. Response-channel fidelity remains modest (0.486 at the deployed horizon), as the revision acknowledges. This is a property of the estimator rather than of the presentation.

  4. Novelty of the identity. I had listed the factorised-policy cross-curvature identity as the main strength. The revision rightly credits it to LOLA; I value the attribution, but it reduces the contribution I had weighed most.

Taken together, these changes substantially alter the paper's theoretical framing and positioning. I believe that version is more accurate and will make a stronger submission, but it deserves a full review of the complete manuscript rather than assessment through a rebuttal thread.

I have no further questions for the authors, and thank them again for their effort.

AuthorsOpenReview ↗

Clarification of the Contribution and Revision Scope (to Reviewer evTx Q2)

Thank you for the careful follow-up and for acknowledging that the scope and disclosure requests have been addressed. We appreciate your candid assessment and offer three brief clarifications about the contribution and the extent of the revision.

1 Continuity with the submitted work

The revision retains the original CoRVM formulation, matched frozen/scripted/co-adaptive interventions, and ExSpectra estimator and allocation method. The numerical results in Tables 1–2 are unchanged. The revision adds requested controls, proofs and implementation details, and narrows the theoretical and empirical claims. These changes refine the scope and evidence for the original framework.

2 Contribution beyond the score-product identity

We credit the score-product identity to LOLA. Our contribution rests on formalizing replay-ranking reversals under teammate learning, defining the scripted intervention that separates reference-learning effects from candidate-induced response, and using this structure for shared replay scoring. The two reversal constructions demonstrate that either channel alone can change the preferred replay item. Eq. (3) remains exact for the specified matched interventions, including the configured learner; accuracy of the scoring approximation is evaluated separately through audits. We agree that this identity does not extend Theorem 1 to deployment.

3 Evidence beyond rank correlation and the scale trade-off

On the original Table 1 panel, the uncalibrated structural score reaches Gap 59.8 versus 45.0 for the strongest replay-priority baseline. On the common calibration panel, separate channels improve over the ExSpectra and JointInf scalar controls by +3.53 [2.88,4.18] and +3.16 [2.38,3.94], respectively (paired 95% CIs over three seed-level means). These comparisons support structural scoring and channel-wise calibration; superiority over MAC-PO+Cal remains unresolved.

The response channel's correlation is 0.486, as noted. Removing it lowers Gap from 68.7 to 59.7 (Table 1) and lowers the reported mean nAUC on 10/11 tasks (Table 2). This provides additional evidence of decision-relevant signal alongside the component-correlation audit.

Since you noted that the revised version is more accurate, we would be grateful if this strengthened and more carefully scoped version could be reflected in your final evaluation. Thank you again for the detailed review.

Reviewer_pdxFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Co-Adaptive Utility Decomposition (CAUD), a value evaluation metric that assesses the importance of experiences by examining their contribution to joint team learning. It further introduces ExSpectra, an experience selection method based on CAUD, to improve the efficiency of experience extraction.

Critical strengths

Strength 1: This paper introduces a novel perspective by highlighting that the value of an experience can change when multiple agents learn collaboratively. This differs from traditional experience replay methods, which focus primarily on the impact of an experience on an individual agent.

Critical weaknesses

Weakness 1: The justification for Co-Adaptive Utility Decomposition focuses primarily on the rationale behind the decomposition, but lacks sufficient theoretical grounding to establish that the three components reliably capture the claimed mechanisms like the complementarity and the induced response.

Questions for authors

Q1: Why did ExSpectra not use the importance sampling as PER does? Does sampling bias affect training? Q2: Why did ExSpectra choose the non-replacement sampling method? Without replacement, although the high-score experience is ensured to be reused, each experience can only be used once. Can this allow the agent to fully learn? Q3: Why did ExSpectra use Co-Adaptive Utility Decomposition to estimate the co-adaptive update utility instead of directly estimating and assessing this value?

4 comments
AuthorsOpenReview ↗

Response to Reviewer pdxF

We thank the reviewer for the constructive review and for recognizing the perspective that replay value changes as teammates learn. Your questions helped us strengthen the mechanism evidence and clarify sampling, repeated replay, and direct estimation. We address each concern below; revisions are marked in blue.

1 Evidence for the complementarity and response mechanisms (W1)

Interventions. After the same replay update, we vary teammate continuation:

BranchTeammate continuation
FrozenHeld fixed
ScriptedOverwritten with the recorded reference-batch path
Co-adaptiveUpdated normally, responding to the candidate

Γcomp\Gamma_{\rm comp} (scripted minus frozen) measures how reference teammate learning changes candidate utility; Γresp\Gamma_{\rm resp} (co-adaptive minus scripted) measures the additional candidate-dependent effect. In H=4H=4 teammate-freeze controls with four active updates, branch states and returns match and both corrections are zero.

Theory. At H=2H=2 under Theorem 1, leading complementarity couples reference teammate displacement with cross-agent return curvature. Response depends on teammate-update sensitivity to active-owned state. If the complete teammate update map is independent of that state, response is zero for any HH. For H≥3H\ge3, complementarity can also involve active update-Jacobian changes.

Each channel can change a decision. Prop. 1 reverses rankings with response zero. Prop. 1b (App. B) gives the converse: complementarity is exactly zero, yet response changes frozen utilities (+.0113,−.0117)(+.0113,-.0117) to co-adaptive utilities (−.0247,+.0020)(-.0247,+.0020). Eq. (6) selects the co-adaptive maximizer in both.

Deployed learner. We audit predictions (c,rc,r) on H=4H=4 RMSProp, beyond Theorem 1's assumptions:

EvidenceComp. (cc)Resp. (rr)
Original audited rank correlation0.6060.486
New same-target correlation margin (c−rc-r / r−cr-c)+0.281+0.141
Original Gap without channel (full: 68.7)61.959.7

Margins subtract predictors' correlations with the same component label (new family-balanced panel). Response remains harder to predict. Without fitting, f+c+rf+c+r reaches Gap 59.8 versus 40.6 for frozen score ff.

2 Importance sampling and training bias (Q1)

Selection changes which TD gradients drive learning. PER's IS targets uniform-buffer loss; ucou^{\rm co} evaluates the selected, unweighted update. Weighting changes that update and its utility. Eq. (8)'s ϵs=0.05\epsilon_s=0.05 uniform mixture gives every remaining candidate positive draw probability. Focal-U shares the pool and focal schedule but selects uniformly. In Table 2 (11 tasks, 8 seeds), task-equal nAUC is 38.16 for Focal-U and 44.49 for ExSpectra (+6.33), with higher means on all 11 tasks. The weighting ablation uses a separate five-task, five-paired-seed panel:

WeightingnAUCDifference: 95% CI
None37.86Reference
Annealed38.21+0.35 [-0.25,0.95]
Clipped, β=1\beta=136.44-1.42 [-1.87,-0.97]

Weights use we∝πe−βw_e\propto\pi_e^{-\beta} with final inclusion probability πe\pi_e, median-based clipping and batch-mean normalization; these variants are not exactly unbiased IS. CIs use paired t over five seed-level task averages. Annealed weighting shows no clear gain; clipped β=1\beta=1 lowers nAUC on this panel.

3 Repeated replay and sampling without replacement (Q2)

Without replacement prevents duplicate slots within one update. Selection does not remove sequences; they remain replayable while in the buffer. Each update selects 32 distinct sequences from 128 candidates, matching Focal-U's budget and Prop. 2's set objective.

4 Decomposition and direct utility estimation (Q3)

Direct measurement supplies sparse audit labels; decomposition shares coefficients across candidates and calibrates channels separately. Each of rr repeats resamples slots and reruns all candidate/reference branches for HH steps: 129r129r branches and 516r516r update positions per allocation (M=128,H=4M=128,H=4). Algorithm C.1 specifies 140 branches and 560 update positions per complete online audit (Rp=10,m=4,H=4R_p=10,m=4,H=4), scheduled every 500 updates if budgets permit. ExSpectra's candidate gradients and derivative refreshes add to these nominal audit counts. On the separate five-task, M=128M=128 bootstrapped truncated-return audit (App. E.9), Gap is 58.08 for MC(1), 64.28 for MC(2), and 69.22 for ExSpectra. The five-repeat cross-fit reference defines 100. Matched calibration also compares direct first-order influence (JointInf):

Fixed 11-task panelGapThree-channel gain: 95% CI
ExSpectra, 3 channels68.23Reference
ExSpectra, 1 scalar64.70+3.53 [2.88,4.18]
JointInf, 1 scalar65.07+3.16 [2.38,3.94]

These are paired t intervals over three seeds. ExSpectra also avoids unrolled differentiation: Table E.4 reports JointInf/ExSpectra per-candidate scoring-time ratios of 1.66-2.53.

Thanks for the detailed rebuttal, which fully addresses my concerns. I am raising my score to 3.

AuthorsOpenReview ↗

Thanking the reviewer pdxF for raising the score

Thank you for your thoughtful engagement throughout the discussion and for raising your score. Your questions directly strengthened the paper, and all clarifications and additional results from this discussion will be retained in the final version. We truly appreciate your time and constructive feedback.

AuthorsOpenReview ↗

A gentle note on the rating update

Thank you again for your support. As a gentle note, the rating displayed in your official review still appears unchanged on our side, possibly because the review form has not been updated yet. If convenient, could you kindly update it before the discussion period ends? We appreciate your help.

Reviewer_QJ5SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies an interesting problem. An experience that is useful for replay now may become more or less useful as teammates continue learning. The proposed method accounts for this interaction, and the extensive experiments, ablations, and resource-matched comparisons provide substantial support.

Critical strengths
  1. The problem is meaningful. The paper recognizes that replay usefulness depends on teammates’ subsequent learning and develops a decomposition and estimator to account for these interactions.

  2. The experimental evaluation is extensive. Experiments across five benchmark families include mechanism analyses, ablations, and resource-matched comparisons, providing substantial evidence for the proposed approach.

Critical weaknesses
  1. The paper introduces many symbols and terms before explaining their purpose. Readers must work through the notation to understand the replay-selection problem. A worked door-opening example showing two candidates, their reference replacements, and a ranking reversal would make the method much easier to follow.

  2. The proposed ExSpectra requires candidate gradients, shared derivative estimates, evaluation rollouts, shadow training, and calibration. Complete training-time measurements are needed to judge whether its performance gains justify the overhead.

  3. Candidates are scored after only a few further updates using predetermined replay data. Longer training also changes agents’ behaviours and the experiences they collect. Consequently, a candidate that looks useful under the short continuation may not remain useful later; the connection needs clearer justification.

  4. A candidate is valued by replacing a sequence in a reference batch and following a prescribed continuation. Its score may therefore depend on both choices. Repeating the comparison with different reference and continuation batches would show whether replay rankings and selected experiences are stable.

Questions for authors
  1. Could you give a step-by-step door-opening example showing how Equation (2) scores a replay experience by replacing a reference experience?

  2. How much extra training time does ExSpectra require compared with the strongest baselines, counting all its additional computations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer QJ5S

We thank the reviewer for the careful review and for recognizing the meaningful problem and extensive evaluation. Your suggestions helped us improve the explanation and test runtime, later learning, and stability more directly. We address each concern below; revisions are marked in blue.

1 A worked example of replay valuation (W1, Q1)

Revised Section 2 explains the problem before notation and defines symbols on first use. Eq. (2) compares candidate replacement against the unchanged reference under the same continuation:

Setting. In Prop. 1's three-state construction (App. B), Agent 1 presses plate A/B and its teammate chooses door A/B. Matching earns +1 and mismatching -1 in the cooperative state. The three-state expected return is J=13tanh⁡(x/β)tanh⁡(y/β)J=\tfrac13\tanh(x/\beta)\tanh(y/\beta), with preferences x,yx,y and softmax temperature β=0.2\beta=0.2. Initially x=0,y=0.1x=0,y=0.1. Candidates eA,eBe_A,e_B record successful A/A and B/B actions.

1. Replace the slot. With B=1B=1, the reference is a zero-gradient experience. Replacing it with eAe_A or eBe_B and updating Agent 1 (η0=0.1\eta_0=0.1) gives x=0.09x=0.09 or −0.11-0.11. The reference keeps x=0x=0 and return zero.

2. Continue learning. At H=2H=2, one prescribed teammate update (η1=0.2\eta_1=0.2) gives y=−0.12y=-0.12 (learns door B) or 0.280.28 (learns door A), depending on continuation data. Freezing teammates keeps y=0.1y=0.1.

3. Subtract reference return. Eq. (2) gives:

Utility / scoreeAe_AeBe_B
Frozen utility+0.065-0.077
Co-adaptive utility, learns B-0.076+0.090
Co-adaptive utility, learns A+0.125-0.148
Eq. (6), learns B-0.060+0.074
Eq. (6), learns A+0.175-0.214

Frozen scoring chooses eAe_A, but learning toward door B makes eBe_B better. The checkpoint is identical across continuations; checkpoint-only scores cannot distinguish them. Eq. (6) uses the continuation direction and ranks correctly in both.

2 Complete training time and benefit (W2, Q2)

We measure full training on 11 tasks with three seeds per method (A100-80GB, eight CPU workers), including candidate/shared derivatives, evaluation rollouts, shadow training, and calibration (App. D.5). This timing panel fixes main-training interaction budgets; Table 2 separately matches total resources.

MethodFull-run time / QMIX
QMIX1.000
MAC-PO1.048
PER+XR1.155
MAC-PO+XR1.207
MARR1.301
JointInf+Cal1.330
ExSpectra1.194

Ratios are averaged within task/seed pairs relative to QMIX: ExSpectra adds 19.4% runtime. Separately, under Table 2's matched total resources, ExSpectra has higher mean nAUC than MAC-PO+XR on 10/11 tasks (+1.54 points on average) and PER+XR on 11/11 (+3.68).

3 Short-horizon utility and later learning (W3)

Full training with fresh collection. ExSpectra exceeds Focal-U (same pool and focal schedule, uniform selection) on all 11 tasks: mean nAUC 44.49 versus 38.16 (Table 2).

Scoring horizon. H=1/2/4/8H=1/2/4/8 yields mean nAUC 38.6/43.2/44.3/43.9 on the five-seed panel (Table E.2). Extending beyond four steps does not improve the mean here.

New fork experiment. We compare ExSpectra and Focal-U from 40% checkpoints: each selects replay for 500 updates, then switches to uniform replay with fresh data collection. Four source seeds per task, two branch seeds per method; main updates matched. App. E.6 also reports w/o CAUD and JointInf+Cal (all JointInf+Cal CIs include zero). ExSpectra minus Focal-U, in each task's return units:

Task500 updates after switch2,000 updates after switch
LBF-3+0.072 [0.040,0.104]+0.150 [-0.069,0.369]
CRing+6.153 [-1.999,14.304]+13.375 [12.113,14.636]

Exploratory paired t 95% CIs use four source-level averages (df=3). LBF-3's means are positive at both times, with the interval excluding zero at 500 updates; CRing's excludes zero at 2,000. This supports an empirical link from local selection to later learning.

4 Stability under reference and continuation redraws (W4)

We redraw reference batches, continuation data, or both: 60 blocks (5 tasks x 4 checkpoints x 3 seeds), 10 redraws per block from the same distributions, fixed 128-candidate pools. Mean pairwise stability of calibrated scores:

RedrawnSpearmanTop-32 overlap
Reference only0.9640.760
Continuation only0.9000.650
Both0.8700.612

Overlap is intersection size / 32; independent random sets give 0.25. A separate simplified real-utility audit (LBF-3, CRing; H=4H=4, 32 candidates, Bref=1B_{\rm ref}=1, top-8) uses 10 repeats per candidate/context, split into halves of five. Mean split-half rank correlation is 0.846. Cross-context correlation of the 10-repeat means is 0.835; top-8 overlap is 0.689. The singleton reference differs from deployment's B=32B=32. Rankings retain substantial agreement under redraws; changing the direction of teammate learning can still reverse them, as in the example. New tests are in App. E; Section 6 clarifies the local-to-long-run scope.

The additional experiments address most of my concerns, particularly the runtime accounting and ranking-stability tests. My remaining concern is the practical trade-off between scoring experiences and spending those resources on learning. Could the authors clarify whether ExSpectra retains its advantage under equal elapsed training time, and in which settings its additional computation is most worthwhile? The new fork experiment is helpful, although it supports persistence of gains from 500 selection updates rather than directly validating individual short-horizon utility scores.

AuthorsOpenReview ↗

Equal-Time Performance, Scoring Trade-offs, and Validation of Learning Gains

We thank the reviewer for the encouraging follow-up and for recognizing the runtime accounting and ranking-stability analyses. We appreciate your distinction between validating short-horizon scores and demonstrating persistent learning gains. To address the remaining practical question, we have analyzed performance at equal elapsed training time and identified where the additional scoring computation is most useful.

1 Equal-time performance

ExSpectra retains an equal-time advantage over JointInf+Cal: time-averaged performance improves by +3.48 (paired 95% CI [2.03,4.92]), and performance at the common cutoff improves by +3.24 [2.14,4.34]. This supports the practical value of shared scoring relative to joint-influence estimation when auxiliary computation is included.

The supplementary panel covers ten tasks, eight seeds and all eleven methods below. For each task/seed, T is the shortest completed runtime across methods. Using checkpoint-level elapsed times from the learning runs, including auxiliary work, we linearly interpolate normalized performance, integrate from 0 to T and divide by T. We also evaluate performance at T.

MethodTime-averaged performancePerformance at T
ExSpectra37.9059.91
JointInf+Cal34.4256.67
FrzInf+Cal36.4260.83
Focal-U36.5863.82
PER37.6163.47
MAC-PO40.0465.21
MAC-PO+XR38.2161.23
MARR39.4363.46
DIFFER38.9564.83
ReLo39.0163.97
ReMERT38.4664.52

Values are task-equal means in normalized performance points. Against MAC-PO+XR and MARR, ExSpectra's time-averaged differences are -0.31 [-1.62,1.00] and -1.53 [-3.08,0.01], with cutoff performance lower by 1.33 and 3.55 points. Paired 95% CIs use eight seed-level task-averaged differences.

At equal elapsed time, ExSpectra improves both metrics over JointInf+Cal, while its benefit relative to additional learning depends on the task and comparator.

2 Where scoring is most worthwhile and how its cost can be adjusted

Coordination Ring provides a favorable example: ExSpectra exceeds MAC-PO+XR/MARR by +1.08/+3.50 in time-averaged performance and +2.47/+6.79 at the common cutoff. On LBF-3, its time-averaged gains are +1.30/+3.73, while cutoff performance is lower by 1.57/2.55; here the benefit is earlier learning progress. On small-4, MAC-PO+XR and MARR achieve higher mean time-averaged performance by 3.22 and 12.17 points, respectively. These results distinguish settings where improved selection repays its computation from those favoring additional learning.

The broader resource-matched experiments provide complementary evidence across five benchmark families. In Table 2, ExSpectra's task-equal mean nAUC exceeds MAC-PO+XR/MARR by +1.54/+1.90, with higher reported means on 10/11 and 7/11 tasks. These controls spend the additional allowance on learning. Together with the equal-time panel, they quantify both the value of selection under matched resources and its practical runtime trade-off.

The audit frequency also provides a direct budget control. In the five-task sensitivity panel (Table 3), increasing the probe interval from 500 to 1000 updates changes mean nAUC from 37.74 to 37.40; at 2000 it is 34.10. The 1000-update setting therefore offers fewer probes with a small observed change in learning performance.

3 Score quality and persistence are tested separately

We agree that the fork experiment tests persistence of gains from 500 selection updates after switching to uniform replay. It complements the direct short-horizon selection audit in RQ3: each block contains 64 candidates, and score-selected top-16 sets are evaluated against held-out, cross-fitted singleton utilities. ExSpectra closes 68.7% of the oracle gap versus 45.0% for the strongest replay-priority baseline.

The audits establish short-horizon selection quality, while the learning curves, horizon sweep and fork experiment provide aggregate evidence of subsequent learning gains.

Together, these experiments support structured replay scoring and identify where its selection benefits justify the additional computation. Thank you again for helping sharpen this practical interpretation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Review Summary

Review Summary

Dear Reviewers, AC, SAC, and PC,

Thank you for the careful reviews and concrete suggestions.

Our paper studies how teammates' subsequent learning changes the value of a replay experience. We formalize Co-Adaptive Replay-Value Misalignment (CoRVM), separate reference teammate learning from candidate-induced response through matched interventions (CAUD), and develop ExSpectra to estimate this local utility before replay allocation.

We appreciate the reviewers' recognition of the meaningful problem (R1, R2), the extensive evaluation across five benchmark families (R1), and the computational usefulness of the factorized-policy cross-curvature identity (R3). The revision credits the prior LOLA derivation of that identity and explains its application to replay scoring.

Summary of revisions. Changes are marked in blue; the individual responses give detailed results and protocols.

[R1] Presentation, complete runtime, later learning, and stability. Section 2 now explains the selection problem before the notation and gives a numerical door-opening example following the replacement, continuation, and return difference in Eq. (2). A new 11-task, three-seed complete-training panel includes candidate/shared derivatives, evaluation rollouts, shadow audits, and calibration: ExSpectra's runtime is 1.194 times QMIX's. Separately, under Table 2's matched total resources, ExSpectra exceeds MAC-PO+XR on 10/11 tasks (mean +1.54 nAUC) and PER+XR on 11/11 (+3.68). We keep the timing and learning panels distinct.

New checkpoint-fork experiments test later learning after a finite selection window followed by uniform replay with fresh collection; the appendix reports all four methods and source-level paired intervals, including inconclusive comparisons. Reference/continuation redraws give score-rank correlations of 0.964/0.900/0.870. A separate singleton-reference utility audit reports split-half and cross-context correlations of 0.846/0.835; its simplified setting is explicitly distinguished from deployment.

[R2] Mechanisms, weighting, repeated replay, and direct estimation. We clarify the frozen, scripted, and co-adaptive interventions, and add a response-only reversal construction to complement the original complementarity-only example. Either channel can change a replay decision in its construction. The configured-learner audits test their predictive correspondence.

We distinguish importance weighting of the learner's TD update from propensity weighting of historical calibration probes, and add a learner-weighting ablation. Without replacement prevents duplicates within one update; experiences remain replayable later. We report nominal branch/update-position counts for direct measurement and sparse auditing, without interpreting them as measured wall-clock savings. The separate Direct-MC accuracy panel uses an explicitly distinct bootstrapped truncated-return target, rather than redefining the main episode-limit objective.

[R3] Theory scope, calibration, reproducibility, and scale. Section 3 distinguishes the exact CAUD identity, Theorem 1's fixed-target, unclipped H=2 SGD expansion, and the truncated scoring surrogate. It states explicitly that the deployed H=4 RMSProp/clip/target-copy learner lies outside Theorem 1. H=4 fidelity is now the primary RQ2 evidence: complementarity/response correlations are 0.606/0.486, with lower response fidelity acknowledged.

Proposition 2 is presented as a conditional allocation bound. We add a centered swapped-item refinement and return-unit diagnostics on 66 exhaustive 8-candidate, 2-item blocks. These estimated-label diagnostics do not certify B=32 deployment. We disclose the eight calibration inputs, unpenalized intercept, historical-only fitting/scaling, feature-freezing procedure, and full continuation/cache/training-clock protocol.

A common calibration panel gives Full 68.23 Gap versus 64.70/65.07 for the two single-scalar controls. Channel-only calibration reaches 67.00; Full's +2.73 over MAC-PO+Cal has an interval including zero. We report that uncertainty. At 20v20, ExSpectra is 0.1 Gap and 0.8 nAUC below JointInf+Cal, with paired intervals including zero, while JointInf takes 2.53 times as long per candidate. The abstract, introduction, results, and conclusion now state this accuracy-cost trade-off.

The revisions connect each claim to its evidence: local theory motivates the score structure, configured-learner audits evaluate fidelity, and matched-resource training tests learning benefits. We have narrowed the theory and scaling claims and disclosed the calibrated-baseline comparisons. Thank you again for helping make the paper clearer and more reproducible.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.