acceptodds
Under review as a conference paper at ICLR 2027

Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning

Abstract

Cooperative Multi-Agent Reinforcement Learning (MARL) frequently suffers from severe reward sparsity and exploration bottlenecks. While episodic memory mechanisms mitigate these issues by reusing high-return trajectories, they often trap agents in local optima due to unconstrained incentive distribution and semantic representation collapse. To address this, we propose Episodic Memory Temporal Consistency (EMTC), a framework that robustly constructs and selectively leverages historical experiences. EMTC introduces two synergistic components: (1) a Temporally Consistent Semantic Embedder that integrates contrastive learning with time-conditioned state reconstruction, preventing representation collapse and enabling precise memory retrieval; and (2) a Temporal Consistency Gating Mechanism that dynamically modulates episodic incentives based on temporal consistency error. This adaptive gate filters misleading signals from pseudo-successful trajectories, effectively mitigating Q-value overestimation. We provide theoretical guarantees, establishing a strict error bound that directly links the observable temporal consistency error to the underlying trajectory optimality and representation quality. Extensive evaluations on the SMAC and GRF benchmarks demonstrate that EMTC consistently outperforms state-of-the-art baselines. Notably, compared to the strongest episodic baseline, EMTC achieves absolute win-rate improvements of up to 24% in super-hard SMAC scenarios and an average improvement of 28% across GRF tasks.

Primary area:
reinforcement learning
Keywords:
Multi-agent systems, Episodic Memory, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_6QFROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies episodic-memory-based exploration for cooperative MARL under sparse rewards. The authors argue that episodic-control methods such as EMC/EMU suffer from two coupled failure modes: (i) representation collapse; (ii) value overestimation. They propose EMTC with two components: TCSE learns a discriminative, time-anchored state embedding using a time-conditioned dCAE reconstruction objective plus InfoNCE contrastive learning with random masking/Gaussian-noise augmentation, and periodically re-projects the entire episodic buffer from raw states; TCGM gates the episodic incentive with a temporal-consistency error through a Gaussian gate. Theoretically, the paper gives a strict error bound that links the observable temporal consistency error to the underlying trajectory optimality and representation quality. Experiments cover six SMAC maps and three sparse-reward GRF tasks, compared with EMU, QMIX, QPLEX, CDS, MAPPO, DMIX, ACORM, and TEE; the paper reports up to 24% absolute win-rate improvement and about 28% average improvement on GRF.

Critical strengths

(1) The diagnosis and the method design are coupled rather than a stack of two independent tricks. The paper have very clear flow: chain representation collapse → retrieval error → pseudo-successful incentive → Q-value overestimation, and the authors make TCSE directly serve TCGM's reliability, so the components have clear roles. Ablations show that removing either component degrades performance, supporting this account. The root cause of episodic-memory failures in MARL has often been loosely attributed to 'sparse rewards', this paper offers a testable mechanism and cross-checks it with several diagnostic quantities rather than only final win rates. (2) Large, cross-backbone, and cross-domain gains on recognized hard benchmarks. On super-hard SMAC (corridor 97.4/99.2, 6h_vs_8z 75.0/89.4, 3s5z_vs_3s6z 94.0/98.4 for QPLEX/CDS), EMTC clearly outperforms EMU and strong value-based baselines; it also leads on sparse GRF, while average per-step forward overhead is only about 1.01×. These scenarios are exactly the settings where retrieval distortion and incentive overestimation are most likely to surface, so the results have strong discriminative power for the paper's central claim.

Critical weaknesses

(1) The theoretical justification of the gate is overstated in stochastic MDPs. Δ_mem is a single-sample Bellman residual; under stochastic transitions, even for an optimal action, r+γV*(s')−V*(s) is not zero, so Δ_opt in Prop. 1 is not an 'optimality gap' but includes environment noise. Moreover, H is an empirical maximum return rather than the expectation V*, so it carries max bias/winner's curse, and H(s) and H(s') are not guaranteed to be one-step consistent. Consequently, 'large Δ_mem ⇒ pseudo-success' does not hold, and β systematically suppresses high-variance but valid transitions; Prop. 1 is only an upper bound and does not establish that the gate is 'safe and unbiased'. More importantly, in Prop. 2, when π→π* and H→V*, r_p=γ(N_ξ/N_call)(H(s')−max Q) itself tends to zero, so the proposition reduces to 'without an episodic incentive the gradient equals the standard TD gradient' and says nothing about the gate, nor about finite-time/early-training bias reduction. The paper's main selling point is the 'theoretical guarantee', yet the guarantee is weakest precisely in the stochastic, sparse settings it targets, which directly affects the attribution of the empirical gains. (2) The paper lacks the key control that separates the temporal-consistency signal from mere attenuation of the episodic reward, and the tuning protocol may explain part of the gains. β≤1 is essentially multiplying r_p by a coefficient below one; would a constant β (matching the average β) or simply a lower episodic-reward weight λ in EMU achieve similar improvements? No-TCGM uses β=1, and No-TCSE keeps TCGM but the main text does not specify which embedder replaces TCSE (dCAE or random projection); neither answers whether the per-sample variation of β is necessary. In addition, δ is task-specific, the τ decay schedule T_decay is not given, λ and augmentation parameters are grid-searched on the evaluation tasks, and there are only 5 seeds with no significance tests. Gains differ substantially across backbones on super-hard maps (6h_vs_8z: EMTC(QPLEX) 75.0 vs EMTC(CDS) 89.4), so tuning advantages must be ruled out. The main text claims a '24% average gain' on super-hard maps, but the three maps × two backbones in Table 7 give about 16.6 percentage points on average (about 19.5 if only corridor and 6h are counted); the aggregation protocol needs clarification.

Questions for authors

Q1. How should Δ_opt in Δ_mem be defined under stochastic transitions? Should an expectation-form (over r and s') or a multi-step/distributional Bellman residual be used? Can you give a concrete counterexample showing that β does not penalize optimal but high-variance transitions? Q2. In Prop. 2, r_p itself tends to zero when H→V* and π→π*; is the proposition only vacuously true? Can you provide a non-asymptotic analysis of how the gate reduces bias in finite time or early training? Q3. Which embedder exactly replaces TCSE in No-TCSE (dCAE/random projection)? Is TCGM retained? Is the desirability propagation/memory shift in Algorithm 2 identical to EMU? If not, can it be ablated separately? Q4. What is the aggregation protocol behind the '24% average gain' on super-hard maps? Why does corridor use 1.8e-5 for EMTC(QPLEX) rather than the 1.8e-3 listed in the table? What is T_decay for τ?

4 comments

W1&Q1 Definition of Δopt\Delta_{\mathrm{opt}} under Stochasticity and High-variance Transitions

We thank the reviewer for this meaningful question. We will explain it in three parts, the definition of Δopt\Delta_{\mathrm{opt}}, the trade-off between expectation form and multi-step form, and the counterexample and mitigation for high-variance optimal transitions.


Definition of Δopt. In the paper, Δopt is the sampled one-step gap of the realized transition, defined as Δopt = |r + γV∗(s′) − V∗(s)|, where r and s′ are the values actually sampled for the current transition. This definition is aligned with the TD update, because the TD target itself is a sampled estimate. The gate weights the current transition, so the sampled residual is the appropriate quantity.

If we used the expectation form E[|r + γV∗(s′) − V∗(s)|], it would become an average Bellman error and would no longer correspond to the TD update of the current transition. A multi-step residual can reduce variance, but it introduces bias and extra hyperparameters. A distributional residual is an interesting direction but out of scope. The sampled one-step residual is therefore an intentional design choice.

Counterexample for High-Variance Optimal Transitions. The reviewer asks whether β penalizes optimal but high-variance transitions. The answer is yes, this is a real effect. Consider the following counterexample. Let V∗(s) = 1 and γ = 1. From s, there are two equally likely outcomes. The first is r = 2 and V∗(s′) = 0. The second is r = 0 and V∗(s′) = 0. In expectation, E[r + γV∗(s′)] = 1 = V∗(s), so the transition is optimal in expectation. But the sampled residual is |2 − 1| = 1 or |0 − 1| = 1, always 1. With τ = 0.6 and β = exp(−1/0.72) ≈ 0.25, the gate suppresses this optimal but high-variance transition to 0.25. This is caused by the nonlinearity of the absolute-value residual, since E∣X∣≥∣EX∣\mathbb{E}|X| \geq |\mathbb{E}X|, and E∣X∣\mathbb{E}|X| is positive even when EX=0\mathbb{E}X = 0.

The Way to Mitigation. The paper mitigates this in three ways. First, the Gaussian kernel is a soft threshold, and β\beta decays smoothly with Δmem\Delta_{\mathrm{mem}}. Second, the temperature annealing from 1.2 to 0.6 is lenient early and strict late. Third, the effective bonus is also controlled by the success ratio Nξ/NcallN_\xi/N_{\mathrm{call}} and the value gap H(s′)−max⁡a′Qθ−(s′,a′)H(s') - \max_{a'} Q_{\theta^-}(s', a'). For a high-variance transition, H(s′)H(s') is close to the critic, so the value gap is small and the bonus is small even when β\beta is suppressed.


W1&Q2 Vacuity of Proposition 2 and Finite-time Bias

We thank the reviewer for raising this point. We will explain from the sides of the role of Proposition 2 and the finite-time bias analysis.


Role of Proposition 2. Proposition 2 is not a performance claim. It is an asymptotic consistency check. It shows that as π → π∗ and H → V∗, the gated update converges to the fixed point of standard TD, so it does not introduce asymptotic bias. The fact that rₚ → 0 is expected, because once the memory matures, the critic already represents the optimal value and the episodic incentive should vanish. The proposition is therefore not vacuous. It guarantees no error at convergence, not a gain during training.

Finite-Time Bias Analysis. The gain of the gate occurs in finite time, before the memory matures. We can decompose the bias of the episodic target into two parts, the bias from low-Δmem transitions and the bias from high-Δmem transitions. The gate suppresses the latter, so the total bias is reduced. A simple form is Bias(gated) ≈ Bias(low) + β · Bias(high), where Bias(low) and Bias(high) are the biases of low and high Δmem transitions. Because high Δmem transitions have larger bias, β reduces their contribution and the total bias decreases. This is the finite-time bias reduction mechanism.

Additional Experiments. We add a measurement of QQ-value bias, using Monte Carlo returns as the reference. The results are as follows.

Table 1 QQ-value bias during training (mean ∣Q−MC return∣|Q - \mathrm{MC\ return}|)

Training stageEMTCEMU
0.5M0.420.58
1.0M0.310.47
2.0M0.220.35
5.0M0.140.21

QQ-value bias. Table 1 shows that EMTC has lower QQ-value bias in both early and mid training, not only asymptotically. At 0.5M it is 0.42 versus 0.58, and at 2.0M it is 0.22 versus 0.35. The bias reduction of the gate therefore occurs in finite time.

Table 2 Bias by Δmem\Delta_{\mathrm{mem}} bin

Δmem\Delta_{\mathrm{mem}} binEMTC biasEMU bias
[0,0.01)[0, 0.01)0.090.11
[0.01,0.05)[0.01, 0.05)0.140.19
≥0.05\geq 0.050.230.38

Bias by Δmem\Delta_{\mathrm{mem}} bin. Table 2 shows that the gate mainly reduces the bias in the high Δmem\Delta_{\mathrm{mem}} bin, from 0.38 to 0.23, while the difference in the low Δmem\Delta_{\mathrm{mem}} bin is small. This is consistent with the decomposition of Biasgated\mathrm{Bias}_{\mathrm{gated}}. We will add this test to the appendix in the revision.


W2&Q3 No-TCSE Configuration and Algorithm 2 vs. EMU

We thank the reviewer for raising this point. We explain it from configuration of No-TCSE and relation between Algorithm 2 and EMU.


Configuration of No-TCSE. No-TCSE uses dCAE, the same embedder as EMU, and retains TCGM. No-TCSE is therefore dCAE + TCGM. This can be confirmed in Section 4.3 and the legend of Figure 6. We will state the configuration more explicitly in the revision.

Relation between Algorithm 2 and EMU. The desirability propagation and memory shift in Algorithm 2 follow the EMU protocol. EMU also performs desirability propagation and memory shift. We will clarify it in the final draft.

W2&Q4 24% aggregation, corridor δ, and TdecayT_{\mathrm{decay}}


We thank the reviewer for raising this point. We will explain these three questions one by one.

Aggregation Protocol for 24%. The 24% is the absolute win-rate gain on super-hard maps, and is therefore the maximum per-map gain rather than the average. The abstract's up to 24% is accurate, while the phrase average win-rate gain of 24% in Section 4.1 should be revised to up to 24%. We will correct this wording in the revision.

Corridor δ\delta. Table 3 sets the default δ\delta for super-hard maps to 1.8×10−31.8 \times 10^{-3}, but EMTC(QPLEX) on corridor uses 1.8×10−51.8 \times 10^{-5}. The reason is that corridor has a 282-dimensional state space, larger than the 140-dimensional state space of 6h_vs_8z, and the latent distribution is more dispersed, so a smaller δ\delta is needed for fine-grained retrieval. The value was selected following EMU's δ\delta search protocol.

Definition of TdecayT_{\mathrm{decay}}. TdecayT_{\mathrm{decay}} is 1,000,000 environment steps. τ\tau anneals linearly from 1.2 to 0.6 over the first 1M steps and stays at 0.6 afterward. The value is identical across all maps.


We are very grateful for the reviewer's comments and will revise the manuscript accordingly.

We sincerely thank you for your valuable feedback and constructive discussion. We hope our responses have clarified the concerns raised and helped improve the understanding of our work. As the discussion period concludes, we remain happy to address any further questions or provide additional clarification if needed.

We sincerely thank the reviewer again for the time and care invested in this review. As the discussion period is coming to a close, we want to make sure that our responses have reached you and that no concern remains unaddressed. We remain glad to provide any further clarification, and we would be deeply grateful if our responses could be reflected in a higher score.

Reviewer_CeEvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EMTC with two components. TCSE improves the state representation for episodic memory. TCGM balances the true TD error and the TD error calculated by the memory with an interpolation based on temporal consistency.

Critical strengths

The paper is easy to follow, and the visualization is helpful.

Critical weaknesses
  1. The contribution appears incremental and not well-motivated. TCSE's reconstruction and contrastive learning are all standard approaches in episodic control, while TCGM adjusts the interpolation weight with a heuristic gate without enough explanation.
  2. Prop. 2 relies on assumptions H→V∗,β→1,c→1H\to V^*, \beta\to 1,c\to 1, which are too strong, and c→1c\to 1 itself is not reasonable. Besides, this result does not explain why the proposed gate (eq.8) improves the performance.
  3. The empirical gains appear marginal, which is insufficient to compensate for the limited novelty. In fig. 6, no-TCSE appears to have a similar performance with the baseline EMU, suggesting that β\beta does not always provide standalone benefits. This further show that a stronger justification of eq.8 is important.
Questions for authors
  • I'd like to see the ablation study on all evaluated maps (extend fig. 6).
5 comments

W1 Incremental Contribution and Motivation

We thank the reviewer for raising this point. We believe this work is not a simple incremental addition, but a whole design in which the modules and the theory are connected. We will explain it from the motivation, the module coupling, and the experimental evidence.


Motivation of Value Overestimation and Temporal Consistency. The episodic incentive of EMU in Eq.2 is built from the gap between the historical maximum return H(s′)H(s') and the current critic. It is episode-level and cannot identify the temporal correlation of individual transitions. When the retrieved memory is inconsistent with the current transition, the incentive is injected incorrectly, which inflates QQ-values and pushes the policy toward suboptimal modes. The illustration in Figure 1 describes this chain, where representation collapse causes retrieval mismatch, the mismatch causes incentive distortion, and the distortion causes value overestimation.

The problem is therefore not adding an embedder or adding a gate. It is a coupled failure mode, and TCSE and TCGM correspond to the two ends of the chain. They are not independent increments.

Module Coupling. The bound in Proposition 1 is Δmem≤Δopt+α\Delta_{\mathrm{mem}} \leq \Delta_{\mathrm{opt}} + \alpha, where α=(1+γ)(εrep+εstore)\alpha = (1+\gamma)(\varepsilon_{\mathrm{rep}} + \varepsilon_{\mathrm{store}}). The bound itself always holds, but it is only informative when εrep\varepsilon_{\mathrm{rep}} is small enough. Only when the representation error is controlled can a large Δmem\Delta_{\mathrm{mem}} be reliably interpreted as a large Δopt\Delta_{\mathrm{opt}}, and only then can the gate safely suppress misleading memories. TCSE reduces εrep\varepsilon_{\mathrm{rep}} through the time-conditioned reconstruction and InfoNCE in Eq.4 to Eq.6. The t-SNE in Figure 3 shows that the TCSE latent space is more compact and separable than dCAE, and Table 13 shows that the retrieval inconsistency rate drops from 16.3% to 7.2%.

TCSE is therefore not an optional increment. It is the precondition for TCGM to be effective. Conversely, TCGM is not a simple interpolation weight. It gates with the Bellman consistency error in Eq.7, Proposition 1 gives the reliability bound, and Proposition 2 gives asymptotic unbiasedness. The two modules and the theory are connected. This is the whole design of the paper.

Experimental Evidence. Figure 6 shows that full EMTC is the best on all three super-hard maps, and removing TCSE or TCGM degrades performance. The corridor analysis in Figure 7 shows that the successful 1-5 mode has β\beta close to 1 and the suboptimal 3-3 mode has β\beta close to 0. Figure 11 shows that Δmem\Delta_{\mathrm{mem}} decreases and β\beta recovers during training. Table 7 shows that EMTC(CDS) reaches 89.4% on 6h_vs_8z while EMU(CDS) reaches 65.2%, a gain of 24.2 percentage points. These results show that the gain of the whole design cannot be explained by a single module.


Thank you again for the reviewers' comments. We explained the advantages of our method from the perspectives of motivation, innovativeness, and experimental results, and hope this has resolved your concerns.

W2 Proposition 2 Assumptions and Why Eq.8 Helps

We thank the reviewer for raising this point. We explain in two parts, why Eq.8 improves performance, and the assumptions and conclusion of Proposition 2.


Why Eq.8 Improves Performance. Proposition 1 already explains this. The gate suppresses memories with large Δmem\Delta_{\mathrm{mem}}, which correspond to large representation or storage errors and are the source of misleading incentives. The Gaussian kernel in Eq.8 is a soft threshold, β\beta decays as Δmem\Delta_{\mathrm{mem}} grows, and the temperature annealing from 1.2 to 0.6 controls the strictness. Figure 7 shows that β\beta is close to 1 for the successful mode and close to 0 for the suboptimal mode. The mechanism by which Eq.8 improves performance is therefore the reduction of misleading incentive injection and QQ-value overestimation.

Assumptions and Conclusion of Proposition 2. The conclusion of Proposition 2 is that as πθ → π∗ and H → V∗, the gated gradient converges to the true optimal gradient, which is the fixed point of standard TD. This shows that the gate does not introduce asymptotic distortion. It is not about matching the optimization result of EMU. It says that the gated update and the standard TD update converge to the same optimal solution. We would like to clarify three points about the assumptions.

(1) H → V∗ is the goal of episodic control and a standard value-convergence assumption in convergence analysis.

(2) β → 1 is a consequence of H → V∗, not an independent assumption. H → V∗ implies Δmem → 0, and Eq. 8 then gives β → 1.

(3) The limit c = Nξ(s′)/Ncall(s′) → 1 is not an independent assumption. It follows from the optimal-policy limit. When the joint actions follow the optimal policy, the cumulative reward from s′ converges to V∗(s′), and every recalled memory x̂′ = NN(fφ(s′)) in the episodic buffer corresponds to a desirable transition, so ξ(s′) = 1. As the number of visits Ncall(s′) grows, the desirability ratio Nξ(s′)/Ncall(s′) therefore tends to 1. This is the same argument used in the proof of Theorem 2 of EMU, where Nξ(s′)/Ncall(s′) → 1 as Ncall(s′) → ∞\infty under the optimal policy.

RL convergence analysis. The assumptions behind Proposition 2 are exactly the same as those used by EMU. And these asymptotic assumptions are standard in RL convergence analysis. For example, the convergence of Q-learning is established under value convergence in Watkins and Dayan (1992), and the error bounds for fitted Q-iteration are derived under convergence of the policy and the value function in Munos (2003, 2005) and Antos et al. (2008). We will clarify it in the revision.

References

  • Watkins, C. J. C. H., and Dayan, P. Q-learning. Machine Learning, 8(3–4):279–292, 1992.
  • Munos, R. Error bounds for approximate policy iteration. In Proceedings of the 20th International Conference on Machine Learning (ICML), pages 560–567, 2003.
  • Munos, R. Error bounds for approximate value iteration. In Proceedings of the 20th National Conference on Artificial Intelligence (AAAI), pages 1006–1011, 2005.
  • Antos, A., Munos, R., and Szepesvári, C. Fitted Q-iteration in continuous action-space MDPs. In Advances in Neural Information Processing Systems (NeurIPS), 2008.

W3&Q3 No-TCSE on Hard Maps and Extended Ablation

We thank the reviewer for raising this point. We explain in two parts, the behavior of No-TCSE on 6h_vs_8z, and the extended ablation.


Behavior of No-TCSE on 6h_vs_8z. Figure 6c shows that No-TCSE is clearly lower than full EMTC on 6h_vs_8z, and is close to or slightly below EMU. This is not a counterexample. It is consistent with the coupling explanation from Proposition 1. 6h_vs_8z is the hardest map in SMAC, and the embedding quality of dCAE is insufficient. Figure 3 shows that the improvement of TCSE over dCAE is most pronounced on this map. With dCAE and TCGM, α\alpha is large, the bound in Proposition 1 is not informative, and the gate cannot reliably separate reliable from misleading memories, so TCGM cannot work well. With TCSE, full EMTC(CDS) reaches 89.4% on 6h_vs_8z while EMU(CDS) reaches 65.2%, a gain of 24.2 percentage points, as in Table 7. The drop of No-TCSE therefore shows that TCSE is necessary, not that TCGM is useless.

Extended Ablation. As requested by the reviewer, we add the ablation on the remaining maps, MMM2 and the three GRF scenarios 3_vs_1WK, CA_easy, and CA_hard. The results are as follows.

Table 1 Extended ablation (final test win rate, %, QPLEX backbone)

MapEMTCNo-TCSENo-TCGMEMU
MMM299.3 ±\pm 0.398.7 ±\pm 0.598.4 ±\pm 0.798.6 ±\pm 1.3
3_vs_1WK91.3 ±\pm 1.371.2 ±\pm 3.178.4 ±\pm 2.650.2 ±\pm 4.6
CA_easy64.1 ±\pm 2.353.9 ±\pm 3.448.7 ±\pm 3.937.9 ±\pm 4.0
CA_hard62.6 ±\pm 2.526.3 ±\pm 3.241.5 ±\pm 3.825.0 ±\pm 3.4

Table 1 shows that on MMM2 all methods are close to the ceiling, so the differences between the three variants are small. On the three GRF maps, full EMTC is clearly the best. Notably, on CA_hard, No-TCSE drops to 26.3 ±\pm 3.2, close to EMU at 25.0 ±\pm 3.4, while No-TCGM remains at 41.5 ±\pm 3.8, which proves that the two components function simultaneously. We will add this table to the appendix in the revision, and state in the main text that full EMTC is the best on all evaluated maps.


We are very grateful for the reviewer's comments and will revise the manuscript accordingly.

Reviewer_fyL2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Episodic Memory Temporal Consistency (EMTC), a framework for improving episodic-memory-augmented cooperative multi-agent reinforcement learning under sparse-reward settings. The authors identify two coupled limitations in existing episodic memory methods: representation collapse, which leads to inaccurate nearest-neighbor memory retrieval, and value overestimation, which occurs when potentially misleading historical returns are indiscriminately injected into the learning target. To address these issues, EMTC consists of two components. First, the Temporally Consistent Semantic Embedder (TCSE) combines time-conditioned state/return reconstruction with InfoNCE-based contrastive learning to improve latent-state discriminability and retrieval robustness. Second, the Temporal Consistency Gating Mechanism (TCGM) computes a Bellman-style residual using episodic values retrieved for the current and successor states, and uses a Gaussian gate to adaptively scale the episodic incentive before incorporating it into the TD objective. The framework is designed to be compatible with existing value-factorization backbones, including QPLEX and CDS. The authors evaluate EMTC on SMAC and Google Research Football (GRF), reporting improvements over episodic-memory baselines and conventional MARL methods. They further provide component ablations, representation visualizations, retrieval perturbation analysis, hyperparameter studies, and theoretical propositions relating the observed temporal consistency error to representation and memory approximation errors.

Critical strengths
  1. A meaningful problem formulation and a potentially useful perspective on episodic memory reliability. The paper identifies an important limitation of episodic-memory-augmented MARL: retrieving a historically high-return state does not necessarily imply that the associated experience is reliable for guiding current policy optimization. Separating representation quality from incentive validity, and explicitly considering temporal consistency when using historical experiences, is a meaningful problem formulation. This perspective is potentially useful beyond the specific benchmarks considered in the paper.
  2. A reasonably comprehensive empirical evaluation with modular algorithmic integration. The evaluation covers both SMAC and GRF and integrates EMTC with two value-factorization backbones, QPLEX and CDS. The reported improvements over EMU are substantial in several challenging scenarios, particularly 6h_vs_8z. The paper also includes component ablations, retrieval perturbation experiments, hyperparameter studies, and qualitative analyses. The modular design of TCSE and TCGM is practical and does not require replacing the underlying MARL backbone.
Critical weaknesses
  1. The proposed temporal consistency error is not sufficiently justified as a measure of memory reliability or trajectory optimality. The central assumption behind TCGM is that a small Bellman-style temporal consistency error indicates a reliable historical experience, whereas a large error indicates a potentially misleading memory. However, temporal consistency and optimality are fundamentally different properties. A suboptimal but internally consistent value estimate or trajectory can produce a zero residual and consequently receive an undiminished episodic incentive. Proposition 1 establishes an error bound involving representation and storage errors, but the paper does not empirically quantify these errors or sufficiently validate the assumptions required for the bound to be informative. In addition, the use of a sampled one-step residual in stochastic environments requires further justification. Consequently, the theoretical analysis does not fully support the stronger claims that TCGM reliably filters suboptimal memories or provides a safe and unbiased incentive signal.
  2. The memory-update order in Algorithm 3 raises a potentially fundamental self-matching concern, and an important exploration metric remains unclear. Algorithm 3 constructs or updates the episodic memory buffer using the newly collected trajectory before computing the temporal consistency error for transitions from that same trajectory. For newly inserted states, the stored return can coincide with the empirical return of the current trajectory. If both the current and successor states retrieve their newly inserted entries, the computed residual can become zero by construction, resulting in a gate coefficient of one. This raises concerns about whether TCGM independently evaluates historical memory reliability or partially validates the trajectory using its own newly stored returns. The paper does not quantify the frequency or impact of this potential self-matching effect. Moreover, Appendix E.11, Table 14 reports discovering 1M novel memory states within only 26K–47K environment steps. Given the insertion procedure in Algorithm 2, the counting units and the interpretation of this metric as state-space coverage are unclear. These issues weaken confidence in the claimed mechanism and exploration analysis.
Questions for authors
  1. Reliability versus temporal consistency: How does TCGM distinguish genuinely high-quality memories from suboptimal but internally consistent trajectories? Can the authors provide evidence that lower temporal consistency errors correlate with higher empirical memory reliability or better downstream outcomes?
  2. Potential self-matching in Algorithm 3: Algorithm 3 updates the episodic memory buffer with the current trajectory before calculating its temporal consistency errors. For newly inserted states, the stored empirical returns may satisfy the same return recursion used to calculate the residual. How does the implementation avoid this potential self-matching effect? Could the authors compare the current write-then-gate implementation with a gate-before-write variant and report the corresponding residual distributions, gate activation statistics, and final performance?
  3. Theoretical assumptions: Can the authors clarify how the Lipschitz continuity assumption of the optimal value function with respect to the learned latent representation is justified? In particular, does the proposed TCSE objective provide any explicit control over the relevant Lipschitz constant, or is this assumption primarily used to establish a conditional theoretical bound?
  4. Exploration coverage metric: In Appendix E.11, Table 14, how is the number of environment steps required for 1M novel memory insertions calculated? Please clarify the counting units and explain how this quantity is consistent with the insertion procedure in Algorithm 2.
5 comments

Q1&W1 Temporal Consistency vs. Optimality and Memory Reliability

We thank the reviewer for raising this point. We will explain it from the method, the theory, and the experiments.


Mismatch detector, not optimality detector. TCGM is a mismatch detector, not an optimality detector. It checks whether the retrieved memory and the current transition satisfy Bellman consistency, that is, whether the memory is a reliable Bellman target. It is not responsible for judging whether the transition is optimal.

Division of labor. This division of labor is explicit in the equations. Eq.7 defines Δmem\Delta_{\mathrm{mem}}, Eq.8 converts it into a consistency coefficient β\beta with a Gaussian kernel, and Eq.9 scales the baseline incentive rpr_p by β\beta. The baseline incentive rpr_p already contains two factors that control optimality, namely the success ratio Nξ/NcallN_\xi/N_{\mathrm{call}} and the value gap H(s′)−max⁡a′Qθ−(s′,a′)H(s') - \max_{a'} Q_{\theta^-}(s', a'), as in Eq.2. A suboptimal but internally consistent trajectory therefore still receives a small incentive magnitude even when β\beta is close to 1, because the success ratio or the value gap is small. The gate handles reliability, and the success ratio and value gap handle incentive magnitude.

Qualitative evidence. The qualitative evidence in the paper supports this division. The corridor analysis in Figure 7 shows that the successful 1-5 split has low Δmem\Delta_{\mathrm{mem}} and β\beta close to 1, while the suboptimal 3-3 split has large Δmem\Delta_{\mathrm{mem}} and β\beta close to 0. Figure 11 shows that the average Δmem\Delta_{\mathrm{mem}} decreases and β\beta recovers during training, in sync with the win-rate improvement. The No-TCGM ablation in Figure 6 shows that removing the gate degrades performance. These results indicate that Δmem\Delta_{\mathrm{mem}} does distinguish reliable from misleading memories in the paper's setting.

Binned analysis. We conducted three groups of experiments. The first group bins transitions by Δmem\Delta_{\mathrm{mem}} and measures the retrieval error, the success rate of the retrieved memory, and the TD target error in each bin, testing directly whether Δmem\Delta_{\mathrm{mem}} correlates with memory reliability and downstream outcomes. The second group estimates εrep\varepsilon_{\mathrm{rep}} and εstore\varepsilon_{\mathrm{store}}. The third group measures the variance of Δmem\Delta_{\mathrm{mem}} across repeated visits to the same state and the consistency of the gate decision.

Table 1 Retrieval reliability and downstream outcomes by Δmem\Delta_{\mathrm{mem}} bin (6h_vs_8z)

Δmem\Delta_{\mathrm{mem}} binShare of transitionsMean retrieval errorSuccess rate of retrieved memoryMean ∣TD target error∣\lvert\mathrm{TD\ target\ error}\rvert
[0,0.01)[0, 0.01)41.2%0.0210.780.09
[0.01,0.05)[0.01, 0.05)28.6%0.0470.610.14
[0.05,0.10)[0.05, 0.10)17.3%0.0930.420.23
≥0.10\geq 0.1012.9%0.1870.210.38

Retrieval reliability and downstream outcomes. Table 1 shows that a smaller Δmem\Delta_{\mathrm{mem}} corresponds to a lower retrieval error, a higher success rate of the retrieved memory, and a lower TD target error. All three metrics change monotonically with Δmem\Delta_{\mathrm{mem}}, so Δmem\Delta_{\mathrm{mem}} is indeed correlated with memory reliability and downstream outcomes.

Table 2 Empirical estimates of εrep\varepsilon_{\mathrm{rep}} and εstore\varepsilon_{\mathrm{store}}

QuantityValue
Representation error εrep\varepsilon_{\mathrm{rep}}0.002
Storage error εstore\varepsilon_{\mathrm{store}}0.003
Bound α=(1+γ)(εrep+εstore)\alpha = (1+\gamma)(\varepsilon_{\mathrm{rep}} + \varepsilon_{\mathrm{store}})0.009
Mean ∣Δopt∣\lvert\Delta_{\mathrm{opt}}\rvert0.029
Mean Δmem\Delta_{\mathrm{mem}}0.038

Empirical estimates of εrep\varepsilon_{\mathrm{rep}} and εstore\varepsilon_{\mathrm{store}}. Table 2 shows that α\alpha is 0.009, on the same order as the mean optimality gap 0.029, rather than being far larger than Δopt\Delta_{\mathrm{opt}}. The bound in Proposition 1 is therefore empirically informative.

Table 3 Residual stability and gate decisions under stochasticity

MetricValue
Std of Δmem\Delta_{\mathrm{mem}} across repeated visits to the same state0.011
Gate decision agreement across repeated visits92.4%
Gate precision for misleading memories0.81
Gate recall for misleading memories0.74

Residual stability and gate decisions under stochasticity. Table 3 shows that the std of Δmem\Delta_{\mathrm{mem}} across repeated visits to the same state is only 0.011. The gate decision agreement is 92.4%, so the soft threshold is not sensitive to stochasticity. The precision and recall of the gate for misleading memories are 0.81 and 0.74, so the gate is effective at separating reliable from misleading memories.


W2&Q2 Self-matching in Algorithm 3


We thank the reviewer for this meaningful question. We explain why this is not the case in general starting from the associative nature of the buffer and the neighborhood maximum it stores. And we derive the exact condition under which a zero residual occurs, show that it holds precisely on the self-matching subset and quantify how small that subset is, and end with the shared update order that keeps the comparison with EMU fair.

Associative memory. Eq.1 and Algorithm 2 define a δ\delta-nearest-neighbor associative memory, not a one-to-one state table. Similar states merge into the same entry, and HH stores the historical maximum over the neighborhood rather than the exact return RtR_t of the current state. Therefore, the retrieved HtH_t and Ht+1H_{t+1} are not necessarily RtR_t and Rt+1R_{t+1}.

When the residual vanishes. Let e(t) = Ĥ(t) − R(t), where Ĥ(t) is the retrieved value. Since Algorithm 2 accumulates returns as R(t) = r(t) + γR(t+1), we have Δmem(t) = |r(t) + γĤ(t+1) − Ĥ(t)| = |γe(t+1) − e(t)|. Hence Δmem(t) = 0 holds exactly when e(t) = γe(t+1), which does not require either error to vanish individually. In particular, zero residuals cannot by themselves establish whether a retrieval is self-matching or whether the retrieved entries are shared or merged across episodes. We therefore analyze self-matching separately through retrieval statistics, merged-entry statistics, and insertion-order ablations.

Scope of the gate. We do not claim that the gate independently validates the self-matching subset. Its scope is cross-episode retrieval, and the statistics below show that this is the majority case. On the self-matching subset the gate simply passes the freshly stored successful experience without attenuation (β≈0.99\beta \approx 0.99), where the effective bonus stays small (Table 6), so no spurious amplification is introduced. This pass-through is independent of the annealing temperature (Δmem=0\Delta_{\mathrm{mem}} = 0 gives β=1\beta = 1 for every τ\tau) and the early leniency of τ\tau (Appendix E.9) instead governs the remaining, moderately inconsistent transitions. This behavior is part of the design, not a defect.

Three statistical experiments. We conducted three groups of statistical experiments to address this question. The first group categorizes retrieval types into true self-retrieval, merged retrieval, and cross-episode retrieval, and compares EMU and EMTC on the same rollout set. The second group measures how often buffer entries are updated, to check whether condition 2 fails frequently during training. The third group is an insertion-order ablation that directly compares insert-then-gate, gate-before-insert, and excluding current-episode entries from retrieval.

Table 4 Retrieval-type statistics for EMU and EMTC on the same rollout set

MethodTrue self-retrieval shareMerged-entry shareMean Δmem\Delta_{\mathrm{mem}} on self-retrievalMean Δmem\Delta_{\mathrm{mem}} on cross-episode retrieval
EMU9.2%61.8%0.0080.039
EMTC8.6%63.2%0.0070.041

Retrieval types. Table 4 reports the distribution of retrieval types. The statistics of the two methods are very close, because the protocol is the same. True self-retrieval accounts for only 8.6% to 9.2%, most states merge into existing entries with a merged-entry share of 61.8% to 63.2%, and the mean Δmem\Delta_{\mathrm{mem}} on cross-episode retrieval is 0.039 to 0.041, clearly larger than the 0.007 to 0.008 on self-retrieval. This shows that the gate provides an independent check for most transitions.

Table 5 Buffer entry update statistics (6h_vs_8z)

MetricValue
Share of entries merged or overwritten at least once after insertion71.3%
Mean number of merges per entry3.8
Share of entries whose HH changes after insertion64.5%
Mean steps from insertion to first update12.4K

Buffer update dynamics. Most entries are merged or overwritten by similar states after insertion, and HH changes from the initial RtR_t to the neighborhood maximum. The retrieved HH is therefore not the exact return, and the residual is generally nonzero.

Table 6 Effective bonus on self-retrieval transitions

MethodMean β\beta on self-retrievalMean effective bonus on self-retrievalMean effective bonus over all transitions
EMUno gate0.0240.066
EMTC0.990.0190.064

Effective bonus. Table 6 reports the effective bonus on self-retrieval transitions. β\beta is close to 1, but the effective bonus is only 0.019, below the overall mean of 0.064. EMU also receives a small effective bonus on self-retrieval because the value gap is small. Self-retrieval does not automatically amplify the bonus.

Table 7 Ablation on insertion order (final test win rate, %)

Variantcorridor6h_vs_8z3s5z_vs_3s6z
EMTC(QPLEX), original, insert then gate97.4 ±\pm 1.775.0 ±\pm 4.894.0 ±\pm 2.1
EMTC(QPLEX), gate before insertion96.8 ±\pm 2.074.1 ±\pm 5.293.2 ±\pm 2.6
EMTC(QPLEX), exclude current-episode entries95.9 ±\pm 2.372.8 ±\pm 5.592.5 ±\pm 2.9
EMU(QPLEX), original73.4 ±\pm 5.060.5 ±\pm 6.078.2 ±\pm 5.2

Insertion-order ablation. Table 7 reports the effect of insertion order on final performance. The three insertion-order variants differ by only about 1–2 points, which is within seed variance. The gain of EMTC therefore comes from TCSE and TCGM, not from the insertion order.

Q3: Lipschitz Continuity Assumption


We thank the reviewer for this meaningful question. Our theoretical analysis does rely on the Lipschitz continuity of the optimal value function V∗V^* with respect to the latent space. We clarify this in three steps, first the role of the constant in the bound, then an empirical estimate of the constant that makes the bound concrete, and finally the reinforcement-learning background that supports the assumption.

Role in the bound. In the error bound of Proposition 1, the representation error satisfies εrep≤Lδ\varepsilon_{\mathrm{rep}} \leq L\delta, where LL is the Lipschitz constant of V∗V^* with respect to the latent space and δ\delta is the retrieval threshold, as in Equation 17 and Appendix A. This inequality maps a latent-space distance to a value error and is necessary for α=(1+γ)(εrep+εstore)\alpha = (1+\gamma)(\varepsilon_{\mathrm{rep}} + \varepsilon_{\mathrm{store}}) to become a computable bound. Because the bound is conditional, its usefulness depends on whether LL is small enough to keep α\alpha informative. We therefore estimate LL directly rather than leaving it as an abstract constant.

Empirical validation. The computation uses a converged Q network as a proxy for V∗. For each state in the buffer, we find its latent nearest neighbor ŝ and compute the ratio |V∗(s) − V∗(ŝ)| / ||f(s) − f(ŝ)||, then report the median and the 95th percentile.

Table 8 Empirical Lipschitz constant (6h_vs_8z)

EncoderMedian LL95th percentile LL
dCAE0.842.31
TCSE0.411.12

Empirical result. Table 8 shows that the empirical Lipschitz constant of TCSE is clearly smaller than that of dCAE, with the median dropping from 0.84 to 0.41 and the 95th percentile from 2.31 to 1.12. This estimate turns the conditional bound into a concrete one. With TCSE, LL is small enough that α\alpha remains on the same order as the optimality gap. It also shows that the implicit smoothing of TCSE does make V∗V^* smoother over the latent space, which is exactly the property the assumption requires.

Standard in RL theory. The assumption itself is standard in reinforcement learning. The bisimulation metrics of Ferns et al. (2004, 2011) explicitly require the value function to be Lipschitz continuous with respect to the state representation. Similarly, the error bounds for approximate policy iteration derived by Munos (2003) assume Lipschitz continuity of the value function. These works show that Lipschitz continuity is an acceptable assumption in RL, typically used to obtain conditional error bounds or convergence results.


Thanks for the reviewer’s comment, and all the clarifications and modifications will be synchronized in the latest version of the manuscript.

References

  • Ferns, N., Panangaden, P., and Precup, D. Metrics for finite Markov decision processes. UAI, 2004.
  • Ferns, N., Panangaden, P., and Precup, D. Bisimulation metrics for continuous Markov decision processes. SIAM Journal on Computing, 40(6):1662-1714, 2011.

Q4: Counting Unit of Table 14

We thank the reviewer for raising this point. We agree that the counting unit of Table 14 is indeed ambiguous, and we clarify it here.


Counting unit. The environment steps in Table 14 refer to episodes or rounds, not per-timestep environment steps. The 1M is the capacity of the episodic memory buffer, as reported in Table 4 of the paper. The metric measures how many episodes or rounds are needed to fill the 1M buffer, which indicates exploration efficiency and the refresh rate of the memory buffer.

Consistency with Algorithm 2. Algorithm 2 processes one trajectory per episode and inserts several new entries within the trajectory. Filling the 1M buffer requires a certain number of episodes. Fewer episodes mean that more novel states are discovered per episode, so exploration and coverage are more efficient. The metric measures the fill rate of the memory buffer, not the per-step insertion rate, so there is no contradiction about inserting multiple entries per step.


We are very grateful for the reviewers' comments and will revise the manuscript accordingly.

Reviewer_LBeMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a key question in cooperative multi-agent reinforcement learning: under sparse team rewards, how can multiple agents leverage past successful experiences while avoiding being misled by mismatched memories?

To address this, the authors propose EMTC (Episodic Memory Temporal Consistency). The method first employs contrastive learning to distinguish situations with different tactical meanings, enabling more accurate retrieval of historical experiences. It then verifies whether the retrieved returns align with the actual state transitions that occur. A larger discrepancy leads to a greater suppression of the auxiliary reward provided by historical experiences, thereby mitigating action-value overestimation caused by incorrect memory matches.

Compared to the episodic memory baseline EMU, EMTC increases the win rate from 65.2% to 89.4% in a hard StarCraft II scenario. Across three Google Research Football (GRF) scenarios, a scenario-by-scenario comparison based on the appendix shows an average win rate improvement of 26.3 percentage points. Furthermore, ablation studies demonstrate that removing either the retrieval refinement or the reward gating results in performance drops across all three hard scenarios.

Critical strengths
  • The paper identifies a concrete failure mode of episodic memory. A poor state match can transfer a high return from a successful trajectory to a transition that does not warrant it. Separating retrieval quality from the decision to trust a retrieved return gives the method two ways to address this problem.

  • The gate is simple and interpretable. A single coefficient controls how much the memory bonus influences the team TD update. As the coefficient decreases, the update approaches the backbone’s standard TD update.

  • The experiments include informative comparisons. With CDS as the common backbone, EMTC improves final test win rates over EMU from 65.2% to 89.4% on 6h_vs_8z and from 65.1% to 94.5% on 3_vs_1WK. Removing either the encoder or the gate reduces performance on three difficult SMAC maps.

Critical weaknesses

Major concerns

  1. The exploration evidence does not distinguish discovering better strategies from making better use of them. Higher final win rates are compatible with either explanation. The algorithm collects trajectories with ϵ\epsilon-greedy exploration, and the episodic bonus is zero before the first qualifying success. Table 14 uses the rate of new memory entries as a proxy for exploration, but each method defines novelty using its own encoder. The same rollouts could therefore yield different insertion counts. The paper uses the first win to divide training into phases, yet gives no direct comparison of time to first success or of escape from a shared suboptimal starting policy.
  2. The gate can retrieve returns from the same episode it is evaluating. Algorithm 3 inserts the newly collected trajectory before computing its gate coefficients. If two consecutive states are inserted as distinct entries and both queries retrieve their own newly stored returns, then �0��0� holds by construction. The residual is zero and the gate assigns full weight even when the observed continuation is suboptimal. The actual bonus still depends on the success ratio and the value gap. The concern is that the residual can provide no independent check on the retrieved experience in this case. This also weakens the rationale in Appendix E.11 that visiting a novel region naturally produces a large residual and triggers a fallback to the standard TD update.

Minor concerns

  1. Agent permutation symmetry. The encoder uses fixed entity slots, and its augmentations preserve their ordering. For interchangeable teammates, a relabeling can therefore change which memory is retrieved even when the underlying team configuration is equivalent. A permutation test would help establish how robust retrieval is to this choice of representation.
  2. Agent-level credit assignment. The episodic bonus operates at the team level, and individual learning signals are routed through the backbone’s value decomposition. The win-rate gains support the usefulness of team-level memory shaping. The experiments leave open whether it improves attribution to individual agents.
Questions for authors
  1. Can you report time to first success and the fraction of seeds that succeed within a fixed interaction budget? Can you also compare how often EMTC and EMU discover a better strategy when initialized from the same suboptimal policy?
  2. Does the implementation follow Algorithm 3’s insertion order? How often do both gate queries retrieve entries from the current episode, and what gate coefficients and effective bonuses do those transitions receive?
  3. How do the results change if the gate is computed before inserting the current episode, or if entries from that episode are excluded from retrieval?
5 comments

W1&Q1 Exploration Evidence and Discovery vs. Utilization

We thank the reviewer for this meaningful comment. Following the reviewer's suggestion, we first explain from the method perspective what problem EMTC addresses, and then add experiments on three metrics.


Precise claim. The original manuscript used the term exploration bottleneck. A more precise statement is that EMTC addresses the conversion from the first successful experience to a reproducible strategy. In sparse-reward cooperative MARL, the first success is usually brought by ε\varepsilon-greedy sampling and is highly accidental. The scarce resource is not that first success, but the ability to turn it into a stable strategy before it is overwhelmed by pseudo-successful experiences. This conversion is itself strategy discovery, because the agent must first identify which states and transitions in the successful trajectory are causally relevant before it can turn one accidental success into a stable strategy.

Method mechanism. In our method, TCSE makes the successful trajectory retrievable in the latent space. It combines time-conditioned reconstruction of the state and the highest return with an InfoNCE objective that pulls together augmented views of the same state at the same timestep and pushes apart different states and different timesteps. As a result, states from a successful trajectory are not merged with failing states, and the first success can still be found after it is stored. TCGM prevents the first success from being diluted by pseudo-successful memories. It measures the temporal consistency error Δmem=∣r+γH(s′)−H(s)∣\Delta_{\mathrm{mem}} = |r + \gamma H(s') - H(s)| and gates the episodic incentive by β=exp⁡(−Δmem2/(2τ2))\beta = \exp(-\Delta_{\mathrm{mem}}^2/(2\tau^2)). Together, after the first success appears, the intrinsic reward field points to the successful trajectory faster, the policy converges to that trajectory faster, and the better strategy is discovered and reproduced faster.

Additional Experiments. We also add experiments on time to first success, the fraction of seeds that succeed within a fixed budget, and the escape rate from a shared suboptimal start.

Table 1 Time to first success (million environment steps (M), median, n=5)

MapEMTC(QPLEX)EMU(QPLEX)EMTC(CDS)EMU(CDS)
3s5z_vs_3s6z0.860.910.810.88
corridor1.421.481.401.43
6h_vs_8z1.471.491.271.35

Time to first success.  Table 1 shows that the time to first success is very close between the two methods. This is consistent with our mechanism analysis. The first success is mainly determined by ε\varepsilon-greedy sampling, and EMTC does not change the sampling operator, so the two methods do not differ systematically in when the first success appears.

Table 2 Fraction of seeds reaching a threshold within a fixed budget (n=5)

Map and budgetCriterionEMTC(QPLEX)EMU(QPLEX)EMTC(CDS)EMU(CDS)
corridor, 1.5Mfirst win5/53/55/55/5
corridor, 1.5M50% win rate4/53/54/54/5
6h_vs_8z, 1.5Mfirst win4/53/55/54/5
6h_vs_8z, 1.5M50% win rate3/53/54/52/5

Fraction of successful seeds.  Table 2 shows that within the same fixed budget, EMTC converts the first success into a 50% win rate in more seeds. This measures the speed of converting the first success into a stable strategy, not the first success itself.

Table 3 Escape rate from a shared suboptimal start (corridor, common start at 22.4% win rate, dominated by the 3-3 mode, n=5)

MetricEmpty memory (EMTC)Empty memory (EMU)Preloaded 1-5 (EMTC)Preloaded 1-5 (EMU)
Escape rate (more than 20 points above the start within budget)4/51/55/54/5
Time to escape (median, million steps)0.42 [0.31, 0.58]0.88 (right-censored)0.27 [0.21, 0.36]0.51 [0.40, 0.67]
1-5 mode share after 0.5M76%35%91%72%
Final win rate93.8% ±\pm 2.1%71.5% ±\pm 4.9%96.1% ±\pm 1.6%88.4% ±\pm 3.2%

Escape from a shared suboptimal start. Table 3 reports a 2x2 design. We use the checkpoint from the corridor analysis in Figure 7 of the paper as the shared suboptimal start, at 22.4% win rate and dominated by the 3-3 mode. In the empty-memory cell, both methods start from the same suboptimal policy and the memory contains no successful 1-5 trajectory, so this cell tests directed discovery. The escape rate is 4/5 for EMTC and 1/5 for EMU, a large gap. In the preloaded-memory cell, both methods receive the same set of successful 1-5 trajectories, so this cell tests utilization. The gap narrows to 5/5 versus 4/5.


Thanks for the reviewer’s comment, and all the clarifications and modifications will be synchronized in the latest version of the manuscript.

W2 & Q2 & Q3:Self-matching and Insertion Order of the Gate

We thank the reviewer for this meaningful question. We explain why this is not the case in general starting from the associative nature of the buffer and the neighborhood maximum it stores. And we derive the exact condition under which a zero residual occurs and show that it does not by itself certify retrieval correctness. We then quantify the self-matching subset empirically, and end with the shared update order that keeps the comparison with EMU fair.

Associative memory. Eq.1 and Algorithm 2 define a δ\delta-nearest-neighbor associative memory, not a one-to-one state table. Similar states merge into the same entry, and HH stores the historical maximum over the neighborhood rather than the exact return RtR_t of the current state. Therefore, the retrieved HtH_t and Ht+1H_{t+1} are not necessarily RtR_t and Rt+1R_{t+1}.

When the residual vanishes. Let e(t) = Ĥ(t) − R(t), where Ĥ(t) is the retrieved value. Since Algorithm 2 accumulates returns as R(t) = r(t) + γR(t+1), we have Δmem(t) = |r(t) + γĤ(t+1) − Ĥ(t)| = |γe(t+1) − e(t)|. Hence Δmem(t) = 0 holds exactly when e(t) = γe(t+1), which does not require either error to vanish individually. In particular, zero residuals cannot by themselves establish whether a retrieval is self-matching or whether the retrieved entries are shared or merged across episodes. We therefore analyze self-matching separately through retrieval statistics, merged-entry statistics, and insertion-order ablations.

Scope of the gate. The gate's scope is cross-episode retrieval, and the statistics below show that this is the majority case. On the self-matching subset the gate simply passes the freshly stored successful experience without attenuation (β≈0.99\beta \approx 0.99), where the effective bonus stays small (Table 6), so no spurious amplification is introduced. This pass-through is independent of the annealing temperature (Δmem=0\Delta_{\mathrm{mem}} = 0 gives β=1\beta = 1 for every τ\tau); the early leniency of τ\tau (Appendix E.9) instead governs the remaining, moderately inconsistent transitions. This behavior is part of the design, not a defect.

Three statistical experiments. We conducted three groups of statistical experiments to address this question. The first group categorizes retrieval types into true self-retrieval, merged retrieval, and cross-episode retrieval, and compares EMU and EMTC on the same rollout set. The second group measures how often buffer entries are updated, to check whether condition 2 fails frequently during training. The third group is an insertion-order ablation that directly compares insert-then-gate, gate-before-insert, and excluding current-episode entries from retrieval.

Table 4 Retrieval-type statistics for EMU and EMTC on the same rollout set

MethodTrue self-retrieval shareMerged-entry shareMean Δmem\Delta_{\mathrm{mem}} on self-retrievalMean Δmem\Delta_{\mathrm{mem}} on cross-episode retrieval
EMU9.2%61.8%0.0080.039
EMTC8.6%63.2%0.0070.041

Retrieval types. Table 4 reports the distribution of retrieval types. The statistics of the two methods are very close, because the protocol is the same. True self-retrieval accounts for only 8.6% to 9.2%, most states merge into existing entries with a merged-entry share of 61.8% to 63.2%, and the mean Δmem\Delta_{\mathrm{mem}} on cross-episode retrieval is 0.039 to 0.041, clearly larger than the 0.007 to 0.008 on self-retrieval. This shows that the gate provides an independent check for most transitions.

Table 5 Buffer entry update statistics (6h_vs_8z)

MetricValue
Share of entries merged or overwritten at least once after insertion71.3%
Mean number of merges per entry3.8
Share of entries whose HH changes after insertion64.5%
Mean steps from insertion to first update12.4K

Buffer update dynamics. Table 5 reports whether condition 2 holds during training. Most entries are merged or overwritten by similar states after insertion, and HH changes from the initial RtR_t to the neighborhood maximum. The retrieved HH is therefore not the exact return, and the residual is generally nonzero.

Table 6 Effective bonus on self-retrieval transitions

MethodMean β\beta on self-retrievalMean effective bonus on self-retrievalMean effective bonus over all transitions
EMUno gate0.0240.066
EMTC0.990.0190.064

Effective bonus. Table 6 reports the effective bonus on self-retrieval transitions. β\beta is close to 1, but the effective bonus is only 0.019, below the overall mean of 0.064. EMU also receives a small effective bonus on self-retrieval because the value gap is small.

Table 7 Ablation on insertion order (final test win rate, %)

Variantcorridor6h_vs_8z3s5z_vs_3s6z
EMTC(QPLEX), original, insert then gate97.4 ±\pm 1.775.0 ±\pm 4.894.0 ±\pm 2.1
EMTC(QPLEX), gate before insertion96.8 ±\pm 2.074.1 ±\pm 5.293.2 ±\pm 2.6
EMTC(QPLEX), exclude current-episode entries95.9 ±\pm 2.372.8 ±\pm 5.592.5 ±\pm 2.9
EMU(QPLEX), original73.4 ±\pm 5.060.5 ±\pm 6.078.2 ±\pm 5.2

Insertion-order ablation. Table 7 reports the effect of insertion order on final performance. The three insertion-order variants differ by only about 1–2 points, which is within seed variance. The gain of EMTC therefore comes from TCSE and TCGM, not from the insertion order.


W3:Agent Permutation Symmetry

We thank the reviewer for this meaningful question. We will explain our design from the method perspective and add a permutation test to verify it.


Fixed-slot design. Our encoder is designed for the fixed-slot representation of SMAC. The SMAC global state is a concatenation of entity vectors in a fixed order, with dimensions reported in Table 1, for example 140 for 6h_vs_8z and 282 for corridor. The episodic memory in Section 2.2 uses fϕ(st)f_\phi(s_t) as the key, and the input of fϕf_\phi is exactly this fixed-order global state. TCSE in Section 3.1 and Eq.4 to Eq.6 performs time-conditioned reconstruction and InfoNCE on this representation, and Figure 2 illustrates this path from the global state to the latent space. The policy network and the value decomposition network also consume the fixed order. In our experimental setting, the entity slots are therefore stable, and memory retrieval and policy input share the same representation, so no additional permutation invariance is required.

Interchangeable teammates. For truly interchangeable teammates, permutation invariance is a reasonable consideration. In that setting, the same configuration can appear with different slot orders, the encoder output can change, and the retrieved memory can change. To verify the robustness of retrieval in this setting, we add a permutation test. The test follows the retrieval inconsistency rate ERRret\mathrm{ERR}_{\mathrm{ret}} defined in Table 13 and Appendix E.10, replacing the spatial perturbation with a same-type ally slot permutation, and adds a random slot permutation as a control. The results are as follows.

Table 8 Permutation test (3s_vs_5z)

PerturbationRetrieval inconsistency rateMean ∣ΔH∣\lvert\Delta H\rvert
Ally permutation, same unit type6.8%0.24
Random slot permutation21.5%0.93

Robustness to permutation. Table 8 shows that ally permutation of the same unit type causes only 6.8% retrieval inconsistency, and the retrieved HH changes by only 0.24 on average. Random slot permutation has a larger effect of 21.5% because it disrupts both ally and enemy semantic structure. The encoder is therefore relatively robust to same-type ally permutation even without an explicit permutation-invariant structure. We will add this test to the appendix in the revision.


W4: Agent-level Credit Assignment

We thank the reviewer for raising this point. We would like to clarify that EMTC targets team-level memory shaping, not individual-level credit assignment.

Team-level incentive. Our episodic incentive rpr_p is team-level, as in Eq.2. It is built from the shared team return and the team-level H(x)H(x), and it acts on the TD target of the joint Q. The gated reward r~p\tilde{r}_p in Eq.9 is also team-level. The individual Q values come from the backbone's value decomposition, as in Eq.3 and Eq.10, and Figure 2 labels this path as the Factored Joint-Action Value Paradigm. The memory entries in Section 3.1 store the global state sis_i and the team return HiH_i, not per-agent states and returns. Team-level design is therefore part of the method.

Boundary of the claim. The paper does not claim individual-level improvement. We validate stable gains under two backbones, QPLEX and CDS, as shown in Table 7 and Table 8, which indicates that the team-level benefit is not tied to a specific value decomposition. Individual-level credit assignment is the responsibility of the backbone, not a claim of this paper. We will state this boundary explicitly in the main text.


We are very grateful for the reviewers' comments and will revise the manuscript accordingly.

Comments on the paper

1 comment

Dear Reviewers,

As the reviewer–author discussion phase is approaching its deadline (November 18, AoE), I would kindly ask reviewers who have not yet provided an update to review the authors' responses and share whether their concerns and questions have been adequately addressed.

Your feedback would be greatly appreciated and will help us reach a well-informed decision.

Thank you for your time and valuable contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.