- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies episodic-memory-based exploration for cooperative MARL under sparse rewards. The authors argue that episodic-control methods such as EMC/EMU suffer from two coupled failure modes: (i) representation collapse; (ii) value overestimation. They propose EMTC with two components: TCSE learns a discriminative, time-anchored state embedding using a time-conditioned dCAE reconstruction objective plus InfoNCE contrastive learning with random masking/Gaussian-noise augmentation, and periodically re-projects the entire episodic buffer from raw states; TCGM gates the episodic incentive with a temporal-consistency error through a Gaussian gate. Theoretically, the paper gives a strict error bound that links the observable temporal consistency error to the underlying trajectory optimality and representation quality. Experiments cover six SMAC maps and three sparse-reward GRF tasks, compared with EMU, QMIX, QPLEX, CDS, MAPPO, DMIX, ACORM, and TEE; the paper reports up to 24% absolute win-rate improvement and about 28% average improvement on GRF.
Critical strengths
(1) The diagnosis and the method design are coupled rather than a stack of two independent tricks. The paper have very clear flow: chain representation collapse → retrieval error → pseudo-successful incentive → Q-value overestimation, and the authors make TCSE directly serve TCGM's reliability, so the components have clear roles. Ablations show that removing either component degrades performance, supporting this account. The root cause of episodic-memory failures in MARL has often been loosely attributed to 'sparse rewards', this paper offers a testable mechanism and cross-checks it with several diagnostic quantities rather than only final win rates. (2) Large, cross-backbone, and cross-domain gains on recognized hard benchmarks. On super-hard SMAC (corridor 97.4/99.2, 6h_vs_8z 75.0/89.4, 3s5z_vs_3s6z 94.0/98.4 for QPLEX/CDS), EMTC clearly outperforms EMU and strong value-based baselines; it also leads on sparse GRF, while average per-step forward overhead is only about 1.01×. These scenarios are exactly the settings where retrieval distortion and incentive overestimation are most likely to surface, so the results have strong discriminative power for the paper's central claim.
Critical weaknesses
(1) The theoretical justification of the gate is overstated in stochastic MDPs. Δ_mem is a single-sample Bellman residual; under stochastic transitions, even for an optimal action, r+γV*(s')−V*(s) is not zero, so Δ_opt in Prop. 1 is not an 'optimality gap' but includes environment noise. Moreover, H is an empirical maximum return rather than the expectation V*, so it carries max bias/winner's curse, and H(s) and H(s') are not guaranteed to be one-step consistent. Consequently, 'large Δ_mem ⇒ pseudo-success' does not hold, and β systematically suppresses high-variance but valid transitions; Prop. 1 is only an upper bound and does not establish that the gate is 'safe and unbiased'. More importantly, in Prop. 2, when π→π* and H→V*, r_p=γ(N_ξ/N_call)(H(s')−max Q) itself tends to zero, so the proposition reduces to 'without an episodic incentive the gradient equals the standard TD gradient' and says nothing about the gate, nor about finite-time/early-training bias reduction. The paper's main selling point is the 'theoretical guarantee', yet the guarantee is weakest precisely in the stochastic, sparse settings it targets, which directly affects the attribution of the empirical gains. (2) The paper lacks the key control that separates the temporal-consistency signal from mere attenuation of the episodic reward, and the tuning protocol may explain part of the gains. β≤1 is essentially multiplying r_p by a coefficient below one; would a constant β (matching the average β) or simply a lower episodic-reward weight λ in EMU achieve similar improvements? No-TCGM uses β=1, and No-TCSE keeps TCGM but the main text does not specify which embedder replaces TCSE (dCAE or random projection); neither answers whether the per-sample variation of β is necessary. In addition, δ is task-specific, the τ decay schedule T_decay is not given, λ and augmentation parameters are grid-searched on the evaluation tasks, and there are only 5 seeds with no significance tests. Gains differ substantially across backbones on super-hard maps (6h_vs_8z: EMTC(QPLEX) 75.0 vs EMTC(CDS) 89.4), so tuning advantages must be ruled out. The main text claims a '24% average gain' on super-hard maps, but the three maps × two backbones in Table 7 give about 16.6 percentage points on average (about 19.5 if only corridor and 6h are counted); the aggregation protocol needs clarification.
Questions for authors
Q1. How should Δ_opt in Δ_mem be defined under stochastic transitions? Should an expectation-form (over r and s') or a multi-step/distributional Bellman residual be used? Can you give a concrete counterexample showing that β does not penalize optimal but high-variance transitions? Q2. In Prop. 2, r_p itself tends to zero when H→V* and π→π*; is the proposition only vacuously true? Can you provide a non-asymptotic analysis of how the gate reduces bias in finite time or early training? Q3. Which embedder exactly replaces TCSE in No-TCSE (dCAE/random projection)? Is TCGM retained? Is the desirability propagation/memory shift in Algorithm 2 identical to EMU? If not, can it be ablated separately? Q4. What is the aggregation protocol behind the '24% average gain' on super-hard maps? Why does corridor use 1.8e-5 for EMTC(QPLEX) rather than the 1.8e-3 listed in the table? What is T_decay for τ?
4 comments
Table 2 Bias by bin
| bin | EMTC bias | EMU bias |
|---|---|---|
| 0.09 | 0.11 | |
| 0.14 | 0.19 | |
| 0.23 | 0.38 |
Bias by bin. Table 2 shows that the gate mainly reduces the bias in the high bin, from 0.38 to 0.23, while the difference in the low bin is small. This is consistent with the decomposition of . We will add this test to the appendix in the revision.
W2&Q3 No-TCSE Configuration and Algorithm 2 vs. EMU
We thank the reviewer for raising this point. We explain it from configuration of No-TCSE and relation between Algorithm 2 and EMU.
Configuration of No-TCSE. No-TCSE uses dCAE, the same embedder as EMU, and retains TCGM. No-TCSE is therefore dCAE + TCGM. This can be confirmed in Section 4.3 and the legend of Figure 6. We will state the configuration more explicitly in the revision.
Relation between Algorithm 2 and EMU. The desirability propagation and memory shift in Algorithm 2 follow the EMU protocol. EMU also performs desirability propagation and memory shift. We will clarify it in the final draft.
W2&Q4 24% aggregation, corridor δ, and
We thank the reviewer for raising this point. We will explain these three questions one by one.
Aggregation Protocol for 24%. The 24% is the absolute win-rate gain on super-hard maps, and is therefore the maximum per-map gain rather than the average. The abstract's up to 24% is accurate, while the phrase average win-rate gain of 24% in Section 4.1 should be revised to up to 24%. We will correct this wording in the revision.
Corridor . Table 3 sets the default for super-hard maps to , but EMTC(QPLEX) on corridor uses . The reason is that corridor has a 282-dimensional state space, larger than the 140-dimensional state space of 6h_vs_8z, and the latent distribution is more dispersed, so a smaller is needed for fine-grained retrieval. The value was selected following EMU's search protocol.
Definition of . is 1,000,000 environment steps. anneals linearly from 1.2 to 0.6 over the first 1M steps and stays at 0.6 afterward. The value is identical across all maps.
We are very grateful for the reviewer's comments and will revise the manuscript accordingly.
We sincerely thank you for your valuable feedback and constructive discussion. We hope our responses have clarified the concerns raised and helped improve the understanding of our work. As the discussion period concludes, we remain happy to address any further questions or provide additional clarification if needed.
We sincerely thank the reviewer again for the time and care invested in this review. As the discussion period is coming to a close, we want to make sure that our responses have reached you and that no concern remains unaddressed. We remain glad to provide any further clarification, and we would be deeply grateful if our responses could be reflected in a higher score.
W1&Q1 Definition of Δopt under Stochasticity and High-variance Transitions
We thank the reviewer for this meaningful question. We will explain it in three parts, the definition of Δopt, the trade-off between expectation form and multi-step form, and the counterexample and mitigation for high-variance optimal transitions.
Definition of Δopt. In the paper, Δopt is the sampled one-step gap of the realized transition, defined as Δopt = |r + γV∗(s′) − V∗(s)|, where r and s′ are the values actually sampled for the current transition. This definition is aligned with the TD update, because the TD target itself is a sampled estimate. The gate weights the current transition, so the sampled residual is the appropriate quantity.
If we used the expectation form E[|r + γV∗(s′) − V∗(s)|], it would become an average Bellman error and would no longer correspond to the TD update of the current transition. A multi-step residual can reduce variance, but it introduces bias and extra hyperparameters. A distributional residual is an interesting direction but out of scope. The sampled one-step residual is therefore an intentional design choice.
Counterexample for High-Variance Optimal Transitions. The reviewer asks whether β penalizes optimal but high-variance transitions. The answer is yes, this is a real effect. Consider the following counterexample. Let V∗(s) = 1 and γ = 1. From s, there are two equally likely outcomes. The first is r = 2 and V∗(s′) = 0. The second is r = 0 and V∗(s′) = 0. In expectation, E[r + γV∗(s′)] = 1 = V∗(s), so the transition is optimal in expectation. But the sampled residual is |2 − 1| = 1 or |0 − 1| = 1, always 1. With τ = 0.6 and β = exp(−1/0.72) ≈ 0.25, the gate suppresses this optimal but high-variance transition to 0.25. This is caused by the nonlinearity of the absolute-value residual, since E∣X∣≥∣EX∣, and E∣X∣ is positive even when EX=0.
The Way to Mitigation. The paper mitigates this in three ways. First, the Gaussian kernel is a soft threshold, and β decays smoothly with Δmem. Second, the temperature annealing from 1.2 to 0.6 is lenient early and strict late. Third, the effective bonus is also controlled by the success ratio Nξ/Ncall and the value gap H(s′)−maxa′Qθ−(s′,a′). For a high-variance transition, H(s′) is close to the critic, so the value gap is small and the bonus is small even when β is suppressed.
W1&Q2 Vacuity of Proposition 2 and Finite-time Bias
We thank the reviewer for raising this point. We will explain from the sides of the role of Proposition 2 and the finite-time bias analysis.
Role of Proposition 2. Proposition 2 is not a performance claim. It is an asymptotic consistency check. It shows that as π → π∗ and H → V∗, the gated update converges to the fixed point of standard TD, so it does not introduce asymptotic bias. The fact that rₚ → 0 is expected, because once the memory matures, the critic already represents the optimal value and the episodic incentive should vanish. The proposition is therefore not vacuous. It guarantees no error at convergence, not a gain during training.
Finite-Time Bias Analysis. The gain of the gate occurs in finite time, before the memory matures. We can decompose the bias of the episodic target into two parts, the bias from low-Δmem transitions and the bias from high-Δmem transitions. The gate suppresses the latter, so the total bias is reduced. A simple form is Bias(gated) ≈ Bias(low) + β · Bias(high), where Bias(low) and Bias(high) are the biases of low and high Δmem transitions. Because high Δmem transitions have larger bias, β reduces their contribution and the total bias decreases. This is the finite-time bias reduction mechanism.
Additional Experiments. We add a measurement of Q-value bias, using Monte Carlo returns as the reference. The results are as follows.
Table 1 Q-value bias during training (mean ∣Q−MC return∣)
Q-value bias. Table 1 shows that EMTC has lower Q-value bias in both early and mid training, not only asymptotically. At 0.5M it is 0.42 versus 0.58, and at 2.0M it is 0.22 versus 0.35. The bias reduction of the gate therefore occurs in finite time.