acceptodds
Withdrawn from ICLR 2027

What to Remember for Cooperation: The Content and Timescale of Memory in Sequential Social Dilemmas

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In shared environments, self-interested individuals repeatedly pursue their own objectives, yet locally rational choices can erode cooperation and reduce collective welfare. Sequential social dilemmas (SSDs) provide a formal setting for studying this tension. Existing approaches in SSDs often promote cooperation by modifying incentives or modeling opponents’ learning. A more basic question is whether and under what conditions interaction history itself can support learned cooperation. Here, we augment agents’ current observations with compact summaries of past interactions and systematically examine how memory representation and horizon shape conditional cooperation. Theoretically, we establish sufficient conditions under which a memory-based strategy achieves a strict local return advantage over any memoryless decision rule. Experiments in Coin Game and CleanUp-lite show that memory representations that link outcomes to the agents that produced them improve cooperation. Extending the memory horizon can further improve cooperation, whereas shuffling events within longer memory sequences weakens it.

Primary area:
reinforcement learning
Keywords:
sequential social dilemmas, multi-agent reinforcement learning, cooperation, interaction memory

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_8ou1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the role of memory in sustaining cooperation in sequential social dilemmas, more specifically Coin Game and CleanUp lite as presented in the literature. Additionally, they provide a theoretical argument for one condition where having memory is better than having none.

Critical strengths
  • The paper analyzes the role of memory in cooperation within mixed-motive scenarios. This is an important problem.
  • The paper proposes a very small, clear theoretical case with exact conditions where memory beats memoryless.
  • The paper breaks down the role of different sizes of memory within the agents with Figure 3. It's quite interesting to see that.

On Presentation.

  • It's not another empirical work on LLM inference/behaviour for cooperation, but focuses on more classical multi-agent deep reinforcement learning topics. This is refreshing.
  • The document does not seem largely AI-written to me as a reader, which alleviates the reviewer burden. I thank the authors for their relevant human care. I have not used any AI checker, so I could be wrong.
Critical weaknesses

Novelty and Relevance

The authors claim that most previous works didn't consider the role of memory as a mechanism in sustaining cooperation, but only intrinsic or extrinsic rewards. I think this is an unfair comparison w.r.t. to the previous work since (1) having memory and (2) how you use this memory are better regarded as one capability of the agent itself to sustain cooperation. When you have the capability of remembering nn states in the past, then you may be able to model how much weight the agent should put on different actions, ergo changing the reward. I think changing the reward is quite natural, and I don't see how the memory approach that is being proposed is different from those others. Even the theoretical argument in section 4.3 is at the end about utility.

The training paradigm explained in section 4.2 seems largely the same as the one proposed in ^[1]. I ask the authors to explain more clearly what innovation is proposed here. Is it only the focus on the role of memory? While using a known algorithm?

The main theoretical argument in section 4.3 is uncontextualized. I have not checked the proof in Appendix A. But I have verified that it's not used anywhere and the consequences are not discussed. Intuitively, I would think that the result is not interesting, since there should be cases where you can do strictly better if you have higher capability, as discussed briefly in the first paragraph of this review section.

Presentation

Section 3 with just preliminaries is quite awkward. It states the partially observable n-agent decision game, but as far as I can see, the formalism is not used in the rest of the paper. For example, in the formalism we call agents with I\mathcal{I}, and in the following section we call them with population A\mathcal{A} again. This is signalling low care for the paper.

In line 190, it's unclear why the authors cite Hoppler's social interactions for content and outcome. I retrieved and read that paper, and it looks like they focused on a taxonomy of social interactions. How it relates to content and outcome separation is unclear. Similarly, the accompanying Jurafsky and Gildea citation seems even more irrelevant since their focus is a classical linguistic problem, while here we are in MARL. This strengthens my hypothesis that little care has been invested in editing and proofreading the work.

In line 236, I encounter hc and hd for the first time. What are they? How do you classify cooperative and defective actions? What are these in the games you study? In the equations on line 239, it's difficult to understand in which space the h live. There was no mention I can see in section 3 or other parts of this history.

Figure 2 doesn't say how it should be read (i.e. is higher better for psame?p_{\text{same}}? what is the implication?). So it's difficult to understand who is better and why.

Method

The taxonomy developed in lines 196-199 seems fairly ad hoc, and heavily dependent on the formalization used in line 194. Meaning, if we had another formal model for our state, that naming would make no sense. What is the broader role of having this category? I see then this is what it's used to explain the results, but I don't see why this framing should be interesting to following research.

In Table 1, it's not clear to me what the baseline is. Everything I see is the taxonomy and the methods proposed.

Minor comments:

  • The authors use a partially observable stochastic game, but never say so; instead, in line 147 they use the standard fully observable case. As far as I know, the coin game indeed is fully observable. Claunup-lite too.
  • Table 1 repeats actor only and outcome only in the crosses and checks. This is a very odd presentation choice.

Conclusion

My arguments presented in the novelty section are quite important limitations to the possible impact of the paper. As far as I can see, that drawback is quite central to the paper's main claim, which means that unless I missed some important parts, since I have studied but not directly worked with LOLA, NL and relevant algorithms, I don't think it's easily salvageable. Thus, I recommend a clear rejection.

^[1]: Su, Q., Wang, H., Xia, Y. & Wang, L. A multi-agent reinforcement learning framework for exploring dominant strategies in iterated and evolutionary games. Nat Commun 17, 490 (2025).

Questions for authors
  1. In line 351, it is asserted that it varies non monotonically. Why is it so? A result without any generalizability or use possibility is not very interesting.
  2. For line 172, can you give an example of the same history, the same observation, but different responses? That possibility is unclear to me.
  3. Line 179, how is that state m encoded? What information is present? I guess this needs a lot of design, and shapes what information the network can learn to use.
  4. Was NL trained with population self-play or with mixed-partner (mentor) training? Figure 2
Reviewer_bkk9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the content and timescale of interaction memory affect cooperation in sequential social dilemmas and proposes Memory-Augmented Mixed-Partner Training (MAMPT). MAMPT augments DQN inputs with ordered actor–outcome summaries of recent interaction events and combines population self-play with fixed rule-based mentors to support history-conditioned cooperation while leaving agents’ environment rewards and learning objectives unchanged. In a two-state model, it further derives sufficient conditions under which a memory-conditioned reciprocal response achieves a strict local return advantage over any memoryless decision rule.

Critical strengths

Strengths

  1. This paper examines three aspects of interaction memory: content, horizon, and temporal order. It provides a clear framework for studying history-based cooperation in sequential social dilemmas.

  2. The actor–outcome ablations control the input size, network structure, and training settings. The results show that retaining both actor identity and event outcome improves cooperation more than retaining either one alone.

Critical weaknesses

Weaknesses

  1. The benefit of actor–outcome attribution is largely intuitive. Although the horizon results are interesting, the paper offers limited explanation of when longer memory helps or hurts cooperation.

  2. The main memory analyses rely on DQN. Replicating a core ablation with another learning algorithm would help establish whether the findings extend beyond this setup.

  3. Both environments involve only two players and clearly defined interaction events. It remains unclear how the memory representation scales to multiple partners and more complex interactions.

  4. The theory assumes accurate state identification from history, irreversible deterioration after defection, and a prescribed continuation policy. This local result provides limited guidance on memory design or how reciprocal behavior is learned.

Questions for authors

Please refer to Weakness.

Reviewer_bykAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the content and timescale of interaction memory affect cooperation in sequential social dilemmas. The authors propose Memory-Augmented Mixed-Partner Training (MAMPT), which combines policies conditioned on recent interaction events with training against both co-learning agents and fixed mentors, without changing the learners' environment rewards. The paper also provides sufficient conditions under which a memory-conditioned decision rule has a local return advantage over memoryless rules. Experiments in Coin Game and CleanUp-lite show that memory linking actors to their outcomes supports cooperation, while the useful memory horizon depends on the environment. The results also suggest that preserving the order of events is important when longer histories are used.

Critical strengths

The paper studies an interesting question: what agents should remember, and how much history they need, to support cooperation. Examining the content and timescale of memory provides a useful perspective on cooperation in sequential social dilemmas.

Critical weaknesses
  1. The evaluated tasks are relatively simple, covering only two-agent environments with discrete actions and clear event boundaries. Real-world social dilemmas are much more complex, and it may be difficult to construct the proposed memory representation in such settings. In particular, event detection, actor identification, and outcome interpretation are predefined rather than learned from complex observations, which limits the applicability of the current approach.
  2. The learning setup is limited. Although the paper compares with several cooperation baselines, the proposed memory mechanism is mainly studied with DQN agents trained from scratch. It is unclear whether the findings hold with other RL algorithms or agents initialized from pretrained models, which may already have useful representations and behavioral knowledge.
Questions for authors
  1. How would the method perform with more than two agents? In particular, would the current actor-outcome representation and memory horizon still be sufficient when an agent needs to track the behavior of multiple partners?
  2. Would the same conclusions hold with more advanced RL algorithms or pretrained models? It would be useful to examine whether the preferred memory representation and horizon change under these different learning setups.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.