- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes the role of memory in sustaining cooperation in sequential social dilemmas, more specifically Coin Game and CleanUp lite as presented in the literature. Additionally, they provide a theoretical argument for one condition where having memory is better than having none.
Critical strengths
- The paper analyzes the role of memory in cooperation within mixed-motive scenarios. This is an important problem.
- The paper proposes a very small, clear theoretical case with exact conditions where memory beats memoryless.
- The paper breaks down the role of different sizes of memory within the agents with Figure 3. It's quite interesting to see that.
On Presentation.
- It's not another empirical work on LLM inference/behaviour for cooperation, but focuses on more classical multi-agent deep reinforcement learning topics. This is refreshing.
- The document does not seem largely AI-written to me as a reader, which alleviates the reviewer burden. I thank the authors for their relevant human care. I have not used any AI checker, so I could be wrong.
Critical weaknesses
Novelty and Relevance
The authors claim that most previous works didn't consider the role of memory as a mechanism in sustaining cooperation, but only intrinsic or extrinsic rewards. I think this is an unfair comparison w.r.t. to the previous work since (1) having memory and (2) how you use this memory are better regarded as one capability of the agent itself to sustain cooperation. When you have the capability of remembering states in the past, then you may be able to model how much weight the agent should put on different actions, ergo changing the reward. I think changing the reward is quite natural, and I don't see how the memory approach that is being proposed is different from those others. Even the theoretical argument in section 4.3 is at the end about utility.
The training paradigm explained in section 4.2 seems largely the same as the one proposed in ^[1]. I ask the authors to explain more clearly what innovation is proposed here. Is it only the focus on the role of memory? While using a known algorithm?
The main theoretical argument in section 4.3 is uncontextualized. I have not checked the proof in Appendix A. But I have verified that it's not used anywhere and the consequences are not discussed. Intuitively, I would think that the result is not interesting, since there should be cases where you can do strictly better if you have higher capability, as discussed briefly in the first paragraph of this review section.
Presentation
Section 3 with just preliminaries is quite awkward. It states the partially observable n-agent decision game, but as far as I can see, the formalism is not used in the rest of the paper. For example, in the formalism we call agents with , and in the following section we call them with population again. This is signalling low care for the paper.
In line 190, it's unclear why the authors cite Hoppler's social interactions for content and outcome. I retrieved and read that paper, and it looks like they focused on a taxonomy of social interactions. How it relates to content and outcome separation is unclear. Similarly, the accompanying Jurafsky and Gildea citation seems even more irrelevant since their focus is a classical linguistic problem, while here we are in MARL. This strengthens my hypothesis that little care has been invested in editing and proofreading the work.
In line 236, I encounter hc and hd for the first time. What are they? How do you classify cooperative and defective actions? What are these in the games you study? In the equations on line 239, it's difficult to understand in which space the h live. There was no mention I can see in section 3 or other parts of this history.
Figure 2 doesn't say how it should be read (i.e. is higher better for what is the implication?). So it's difficult to understand who is better and why.
Method
The taxonomy developed in lines 196-199 seems fairly ad hoc, and heavily dependent on the formalization used in line 194. Meaning, if we had another formal model for our state, that naming would make no sense. What is the broader role of having this category? I see then this is what it's used to explain the results, but I don't see why this framing should be interesting to following research.
In Table 1, it's not clear to me what the baseline is. Everything I see is the taxonomy and the methods proposed.
Minor comments:
- The authors use a partially observable stochastic game, but never say so; instead, in line 147 they use the standard fully observable case. As far as I know, the coin game indeed is fully observable. Claunup-lite too.
- Table 1 repeats actor only and outcome only in the crosses and checks. This is a very odd presentation choice.
Conclusion
My arguments presented in the novelty section are quite important limitations to the possible impact of the paper. As far as I can see, that drawback is quite central to the paper's main claim, which means that unless I missed some important parts, since I have studied but not directly worked with LOLA, NL and relevant algorithms, I don't think it's easily salvageable. Thus, I recommend a clear rejection.
^[1]: Su, Q., Wang, H., Xia, Y. & Wang, L. A multi-agent reinforcement learning framework for exploring dominant strategies in iterated and evolutionary games. Nat Commun 17, 490 (2025).
Questions for authors
- In line 351, it is asserted that it varies non monotonically. Why is it so? A result without any generalizability or use possibility is not very interesting.
- For line 172, can you give an example of the same history, the same observation, but different responses? That possibility is unclear to me.
- Line 179, how is that state m encoded? What information is present? I guess this needs a lot of design, and shapes what information the network can learn to use.
- Was NL trained with population self-play or with mixed-partner (mentor) training? Figure 2