acceptodds
Under review as a conference paper at ICLR 2027

CoopShift: Post-Training Amplifies the Memory Curse in LLM Cooperation

97 views

Abstract

How does post-training change cooperation during repeated interaction? Within each of seven released model lineages, we compare pretrained (Base), supervised fine-tuned (SFT), and subsequent preference- or reward-optimized (Aligned) checkpoints across seven social dilemmas. Copies of each checkpoint play one another for 200 rounds, with visible history limited to the last two or forty rounds. The memory curse refers to lower cooperation when agents can see more past rounds. Our central finding is that post-training amplifies this decline. Averaged across the six binary-action games, expanding visible history reduces cooperation at every SFT and Aligned checkpoint. At each post-trained stage, these declines exceed Base's in six of seven lineages. Most of this average amplification is already present at SFT. With long history, both post-trained stages also cooperate less than Base on average in each binary-action game. In the symmetric binary games, their fitted cooperation rates fall more steeply on average than Base's as per-round cooperation costs rise, benefits fall, or noncooperation becomes more rewarding. The difference in cooperation rates between Aligned and SFT also changes from one-shot decisions to repeated play. When given the same past records, their choices depend on whether the instructions describe a final round or possible future interaction. Post-training evaluation should therefore go beyond isolated choices to assess whether cooperation is sustained in repeated interaction and how it depends on payoffs and visible history.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, cooperative AI, post-training, multi-agent, repeated games

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZfSjOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2(−1)
Soundness
2(+1)
Read review
Paper summary

This paper explores how post-training amplifies the memory curse phenomenon in multi-agent LLM cooperation. The authors find that (1) post-training does amplifies the memory curse, i.e., longer memory results in lower cooperation rate, and (2) post-training makes the models more sensitive to changes in payoff.

Critical strengths

S1. This paper explores an interesting research question.

S2. The writing is good and easy to follow.

S3. The experiments are extensive: 21 checkpoints, 7 model lineages, and 7 social-dilemma games.

Critical weaknesses

W1. It has not been answered through what kind of mechanism post-training amplifies the memory curse.

W2. The authors present a testable explanation that post-trained models are less likely to recover cooperation. This implies post-trained models should respond to earlier defection more negatively than the Base checkpoints, which, however, is contrary to the results shown in Table 25.

W3. The paper does not provide any preliminary experiments on the instruction-following capabilities of the Base checkpoints. The authors acknowledge that swapping actions and testing payoff comprehension would be helpful to address this concern.

Questions for authors

See W2.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZfSj (1/3)

We thank the reviewer for recognizing the importance of the research question, the clarity of the writing, and the extensive evaluation across 21 checkpoints, seven lineages, and seven social dilemmas.

W1 · Behavioral pathway: history-conditioned cooperation beyond partner defection

CoopShift's central contribution is to identify how the effect of visible interaction history changes across post-training stages, rather than only whether longer histories reduce cooperation. We compare 21 checkpoints from seven model lineages across seven social dilemmas, with matched Base, SFT, and Aligned evaluations. In Table 3, the mean history effect H = Coop(HL40) − Coop(HL2) across six binary-action games is −1.17 percentage points (pp) at Base, −8.11 pp at SFT, and −10.93 pp at Aligned; the paired SFT-minus-Base and Aligned-minus-Base contrasts are −6.93 pp and −9.76 pp. Most of the average amplification appears by SFT, identifying a key training stage at which history-conditioned cooperation changes. Our payoff and controlled-partner analyses further characterize this effect under different incentives and partner behaviors.

These findings supply both a mechanistic research direction and practical evaluation principles. For agentic post-training, our results motivate assessing (i) whether agents sustain beneficial cooperation as their interaction histories grow, and (ii) whether contributions remain appropriate to payoff incentives and opportunities for reciprocity (Table 2; Appendix U). The concentration of amplification by SFT also motivates evaluating intermediate checkpoints alongside final Aligned ones, including as possible starting points for collaboration- or incentive-aware post-training. A targeted next step is to vary the focal agent's earlier actions, realized payoffs, and partner histories independently while holding the current game state fixed, to identify which history-feedback channels shape later contributions.

To probe the behavioral explanation, we ran a new Always-Cooperative Partner Control to test whether the amplification requires observing partner defection. The focal LLM controls one player in a three-player repeated game while two scripted partners cooperate throughout, including an initial 40-round warm-up. The focal agent still chooses its own actions, and those actions and its realized payoffs enter later histories. We compared HL2 and HL40 across SmolLM2 and DeepSeekMath at Base, SFT, and Aligned stages, six payoff settings spanning PGG, CPR, and Donation-PD, and ten fresh seeds; cooperation is measured in rounds 47–100.

Table R1. Always-Cooperative Partner Control. H = Coop(HL40) − Coop(HL2), averaged over rounds 47–100 and six payoff settings (mean ± sample SD over ten seeds; pp). Both programmed partners cooperate at every round.

CheckpointHistory effect H (pp)
SmolLM2 Base+0.11 ± 0.44
SmolLM2 SFT−19.51 ± 8.10
SmolLM2 Aligned−20.23 ± 5.30
DeepSeekMath Base+0.34 ± 1.46
DeepSeekMath SFT−17.82 ± 4.18
DeepSeekMath Aligned−15.62 ± 1.62

The always-cooperative control sharpens the behavioral explanation. Across both lineages, the two Base history effects are near zero, whereas all four post-trained checkpoints show negative mean effects (−20.23 to −15.62 pp) despite partners never defecting. Thus, observed partner defection and retaliation against it are not necessary for the average amplification in these settings. Because the focal agent's own actions and payoffs still enter its growing interaction record, history-conditioned incentive adaptation is a concrete behavioral pathway to investigate. The finding directs attention to how post-trained agents use accumulated experience when deciding whether to continue contributing, even with reliable partners.

The payoff consequences make this behavior strategically important. In the always-cooperative PGG control (r = 1.2), we compare DeepSeekMath SFT's individual and total group payoffs per round under HL2 and HL40. The longer history coincides with a higher individual return but lower total group return, illustrating why both individual and collective incentives matter alongside cooperation rates.

Payoff metric (points/round)HL2HL40
Evaluated agent's individual payoff1.5791.712
Total group payoff3.4743.429

We will revise Section 4 and the Discussion to foreground this stage-resolved contribution, the behavioral insight from cooperative-partner controls, and the resulting evaluation and future training directions.

AuthorsOpenReview ↗

Response to Reviewer ZfSj (2/3)

(continued)

W2 · Why Table 25 does not contradict the recovery hypothesis

Table 25 was presented as a mechanistic exploration, not as a test of a universal slower-recovery claim. Section 4 proposes a withdrawal–recovery feedback hypothesis: when one agent defects, others may reduce their cooperation in response, potentially reinforcing mutual defection. Appendix J examines a narrower question: whether earlier defections still influence the next decision after a scripted sequence of cooperative rounds. Because those cooperative rounds are provided by the experiment rather than generated through interaction, Table 25 measures sensitivity to older defections, not the process of recovering cooperation.

Importantly, being less sensitive to past defections does not necessarily mean recovering cooperation more easily. For example, a model might (i) withdraw cooperation more strongly when a partner first defects, even if it later becomes less sensitive to that event; (ii) hesitate to cooperate again because it expects the other agent to make the first move; (iii) enter a feedback loop where its own reduced cooperation leads to further unfavorable responses and outcomes, making cooperation increasingly difficult to restore; or various other possibilities. These processes unfold across multiple rounds and cannot be inferred from a single decision after a scripted cooperative history.

Therefore, Table 25 does not logically contradict the proposed withdrawal–recovery hypothesis. Moreover, our new Always-Cooperative Partner Control (Table R1 in our above rebuttal) shows that the amplified history effect can arise even without partner defection, pointing to broader history-dependent behavioral mechanisms. We will clarify the exploratory role of Table 25 in Appendix J and distinguish these potential mechanisms from the directly measured stage-by-history findings in Section 4.

AuthorsOpenReview ↗

Response to Reviewer ZfSj (3/3)

(continued)

W3 · Base model-compatible decision protocol and label-swap controls

Our evaluation is designed to make Base/SFT/Aligned comparisons less dependent on instruction-tuned answer formatting. Instead of asking a model to produce a long rationale and append an exactly formatted action in one response, we use the two-request rationale-to-action protocol (Appendix M.1). This separates generating an explanation from selecting a legal action and avoids extracting decisions from free-form prose.

Phase 1. Generate a rationale. The model receives the game rules, visible interaction history, and legal actions, then is asked to “Briefly state your strategic rationale in at most X words.” It generates ordinary text with a predefined token budget, while no final-answer is required in this phase.

Phase 2. Select an action. A new model request repeats the game and history, includes the generated rationale as an assistant turn, and asks, “Now commit to one action for this round: ” vLLM choice-constrained decoding restricts the final output to that game's legal action labels (e.g., A0/A1 in the PGG template), with an eight-token cap. Importantly, the decoder enforces only a valid response format: it leaves every legal action selectable and does not determine whether the model cooperates. Thus, the cooperation measure comes from the model's semantic action choice, not a parser's interpretation of unrestricted text.

We validated the actual final-action readout. In the audited repeated-CPR Base slice, 336,000 accepted action choices had zero logged final-action parse failures. Rationale retries are tracked separately, and the runner retains the first valid rationale/action rather than inserting a default action. These counts directly address whether Base decisions can be converted reliably into legal actions under the executed interface.

We also tested Base checkpoints on questions with objectively correct answers. Across all these checkpoints, we separated copying, selecting the larger explicitly stated payoff, and calculating a payoff from the rules. On the explicit-payoff comparison, Nemotron-Cascade Base achieved 94.58% and OLMo 3 Base 81.67% correct choices under the constrained readout. Both answer options were legal, so these scores reflect selecting the correct option, not simply satisfying the output constraint. These task-specific tests provide evidence of payoff comparison capability in Base models, complementing the interface audit.

Finally, we tested whether the stage-by-history effect depends on a particular action label. In the label-swap controls, the meanings assigned to legal labels were exchanged, and cooperation was scored by the underlying action, not the token string. For each checkpoint, define H = Coop(HL40) − Coop(HL2), and the within-lineage contrast K = H(post-trained) − H(Base). A negative K means longer histories have a more negative effect after post-training than at Base. Table R2 compares the original lineage-level K values and new label-swap estimates.

Table R2. Stage-relative history contrast K (mean ± reported SD; percentage points).

Model lineagePaper Original K (SFT − Base)Paper Original K (Aligned − Base)Label swap K (SFT − Base)Label swap K (Aligned − Base)
SmolLM2−16.2 ± 2.4−18.1 ± 2.4−8.15 ± 2.27−8.10 ± 1.88
Nemotron-Cascade−4.9 ± 2.9−13.7 ± 3.8−6.73 ± 6.51−27.22 ± 6.02
OLMo 3−15.2 ± 2.8−10.7 ± 2.0−15.92 ± 19.82−35.78 ± 15.97

All six new label-swap mean K contrasts are negative across SmolLM2, Nemotron-Cascade, and OLMo 3, matching the direction of their original lineage-level contrasts. Taken together, the two-stage interface, verified action readouts, task-specific Base capability checks, and action-label controls make the stage-by-history comparisons interpretable without assuming that Base checkpoints have instruction-tuned response-formatting behavior. We will make the executed two-request protocol and audit denominators explicit in Appendices M/N and add the capability and label-swap controls to the final version of the paper.

Thank you again for your thoughtful feedback. We hope our new results and clarifications address your concerns, and would greatly appreciate it if you could consider revisiting your rating.

Reviewer_ZEBkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how post-training affects cooperation in LLMs during repeated interactions, focusing on the amplification of the memory curse. The authors evaluate 21 checkpoints across seven model lineages and seven social dilemmas, comparing Base, SFT, and Aligned models. Through controlled experiments involving varying interaction histories, payoff conditions, and one-shot versus repeated interactions, the study demonstrates that post-training generally amplifies cooperation declines when longer histories are visible. The findings are supported by systematic experimental comparisons and quantitative analyses, offering valuable insights into evaluating cooperation in multi-agent AI systems.

Critical strengths

The paper offers valuable insights into how post-training amplifies the memory curse in LLM cooperation. By systematically comparing Base, SFT, and Aligned checkpoints across model lineages, the study advances existing research and clarifies how cooperative behaviour develops throughout training. The evaluation of 21 checkpoints across seven model lineages and seven social dilemmas, under varied payoff conditions, interaction histories, and repeated-play settings, provides strong empirical support for the findings.

Critical weaknesses

The paper presents a rigorous experimental design and well-supported findings. However, potential prompt-length effects and variations across model lineages warrant further clarification.

The abstract begins directly with the research question without sufficiently establishing the broader context or motivation. Could the authors clarify which specific gap in existing research on LLM cooperation and post-training this study addresses, and why it matters for multi-agent AI systems? Given that prior research has documented the memory curse, could the authors more explicitly distinguish, in the abstract, the established phenomenon from this study's novel contribution? In particular, could the abstract clarify what new understanding is gained by examining cooperation across Base, SFT, and Aligned checkpoints beyond existing findings? The abstract presents numerous detailed experimental findings but does not clearly articulate their broader implications. Could the authors better synthesize the key findings and explain what the observed amplification of the memory curse implies for the design, evaluation, and reliability of post-trained LLMs in multi-agent environments? While the introduction is well structured and clearly motivates the study, could the authors more explicitly articulate the specific research gap and its significance relative to existing work on the memory curse? Sections 3–5 present a systematic, well-supported experimental analysis with clear comparisons across training stages, payoff conditions, and interaction settings. However, given the observed variations across model lineages and games, and the potential influence of prompt length on history-based evaluations, could the authors clarify how these factors affect the interpretation and generalizability of their findings? The discussion and conclusion provide a balanced interpretation of the findings, particularly in distinguishing individual decision quality from collective cooperation and acknowledging unresolved mechanisms. However, could the authors clarify how well findings from homogeneous self-play in structured social dilemmas generalize to real-world multi-agent AI systems with heterogeneous agents and more complex interactions?

Questions for authors

The abstract begins directly with the research question without sufficiently establishing the broader context or motivation. Could the authors clarify which specific gap in existing research on LLM cooperation and post-training this study addresses, and why it matters for multi-agent AI systems? Given that prior research has documented the memory curse, could the authors more explicitly distinguish, in the abstract, the established phenomenon from this study's novel contribution? In particular, could the abstract clarify what new understanding is gained by examining cooperation across Base, SFT, and Aligned checkpoints beyond existing findings? The abstract presents numerous detailed experimental findings but does not clearly articulate their broader implications. Could the authors better synthesize the key findings and explain what the observed amplification of the memory curse implies for the design, evaluation, and reliability of post-trained LLMs in multi-agent environments? While the introduction is well structured and clearly motivates the study, could the authors more explicitly articulate the specific research gap and its significance relative to existing work on the memory curse? Sections 3–5 present a systematic, well-supported experimental analysis with clear comparisons across training stages, payoff conditions, and interaction settings. However, given the observed variations across model lineages and games, and the potential influence of prompt length on history-based evaluations, could the authors clarify how these factors affect the interpretation and generalizability of their findings? The discussion and conclusion provide a balanced interpretation of the findings, particularly in distinguishing individual decision quality from collective cooperation and acknowledging unresolved mechanisms. However, could the authors clarify how well findings from homogeneous self-play in structured social dilemmas generalize to real-world multi-agent AI systems with heterogeneous agents and more complex interactions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZEBk (1/3)

We thank the reviewer for recognizing the systematic checkpoint and game comparisons, and for the constructive suggestions on motivation, interpretation, and broader relevance. We will make the paper's central contribution and its implications for real-world applications more explicit.

Research contribution and implications for agentic post-training

The central contribution of CoopShift is to show that post-training reshapes how LLM agents respond to longer interaction histories in repeated cooperation. Previous work established that exposing agents to more past interactions can reduce cooperation (Liu et al., 2026), but did not establish how that effect changes from Base to SFT to later Aligned checkpoints. Our matched, within-lineage evaluation spans seven model lineages and seven social dilemmas. In the six binary-action games, the mean HL40-minus-HL2 cooperation change is −1.17 pp at Base, −8.11 pp at SFT, and −10.93 pp at Aligned (Table 3). The history effect becomes more negative relative to Base in six of seven lineages at both post-trained stages, with most of the average amplification already present by SFT. This identifies an important stage of model development at which long-horizon cooperative behavior changes, rather than merely another instance of the memory curse.

These findings provide concrete guidance for developing and evaluating more cooperative AI agents. They motivate two evaluation principles for agentic post-training: (i) test whether agents sustain mutually beneficial cooperation as their interaction histories grow, and (ii) assess whether their contributions remain appropriate under different payoff incentives and opportunities for reciprocity. The first follows from our stage-by-history findings; the second is supported by our payoff-response analysis (Table 2) and controlled partner-policy comparisons (Appendix U), which demonstrate why contribution decisions depend on incentives and partners' responses. The results also motivate studying intermediate SFT checkpoints alongside final Aligned models (and other checkpoints in between if available) as possible starting points for collaboration- or incentive-aware post-training. We will make this research gap, the development implications, and these evaluation criteria explicit in the Introduction and Discussion.

Practical significance and real-world applications

Seven model lineages and seven social dilemmas give our checkpoint-by-history analysis broad coverage of model development paths and strategic incentives. Homogeneous self-play makes the comparison across checkpoints controlled: every seat uses the same checkpoint in a given trajectory, so the effect of choosing that stage can be evaluated at the group level.

The same incentive structures arise when agents have different owners, roles, and tools. Public Goods represents costly effort for shared tests or transaction verification, Common-Pool Resource represents restraint when using shared compute or API capacity, Donation-PD represents assistance that can be reciprocated, Assurance represents a joint migration that succeeds only with sufficient participation.

Recently-emerging agent commerce provides concrete examples: CarEdge offers a consumer-facing AI Buying Agent (https://cs2.caredge.com/live) and dealership-facing AI products (https://caredge.com/for-dealers), and Mark Miller Subaru (https://www.markmillersubarumidtown.com/blog/ai-agent-car-sale) has reported an agent-to-agent vehicle transaction involving a CarEdge buyer agent and a dealership-built agent. Luminance (https://www.luminance.com/press-releases/luminance-showcases-worlds-first-completely-ai-powered-contract-negotiation) has demonstrated AI agents negotiating a contract on behalf of opposing parties, while Pactum (https://pactum.com/campaigns-agents) develops agents that negotiate commercial terms with suppliers.

AuthorsOpenReview ↗

Response to Reviewer ZEBk (2/3)

(continued)

These interactions create opportunities for cooperation even when the agents' objectives differ. For example, buyer and seller agents may disagree on price, but both benefit when transaction details are accurate and errors are resolved efficiently. One agent might spend additional time checking quoted fees, correcting transaction information, or resolving a delivery issue. That effort consumes its own resources while creating benefits for both parties, introducing a strategic choice between contributing and relying on the other party's effort. In recurring supplier relationships, such choices can also influence future assistance and reciprocity. Reputation adds another dimension: CarEdge's Dealer Transparency Index (https://caredge.com/rating-methodology) uses historical quotes to evaluate pricing transparency, illustrating how earlier interactions can inform future counterparties. CoopEval similarly studies how repeated interaction and reputation can support cooperation among LLM agents (Tewolde et al., 2026).

The practical implication is a deployable evaluation principle: test checkpoint choice together with visible history and the actual contribution costs, partner responses, and benefits relevant to the workflow. Our controlled comparisons make the stage-by-history effect measurable. Systems with heterogeneous agents can extend the same tests by varying the roles, tool access, information, and partner policies.

We will strengthen the Introduction and Discussion with these concrete practical implications of the games and explain how to evaluate both individual returns and shared outcomes across interacting agents. This connects the empirical result to the reliability of agentic systems in which cooperation matters over repeated decisions.

Controlling initial prompt length in 200-round interactions

To test whether the checkpoint-by-history result is explained simply by the extra initial prompt tokens shown at HL40, we ran 200-round prompt-length-matched self-play. We evaluated complete Base/SFT/Aligned checkpoint sequences for SmolLM2 and Nemotron-Cascade, using PGG (r = 1.2) and CPR (θ = 0.20).

For each lineage, we compared HL2 and HL40 with and without initial-length matching, using five paired seeds. In the length-matched condition, neutral archive text equalized the initial prompt token count before every rationale request.

For each checkpoint, define H = cooperation(HL40) − cooperation(HL2) and K = H(post-trained) − H(Base). Negative K indicates a more negative history effect after post-training. The table reports mean ± SD over five runs or paired seeds: the first column reproduces the original six-binary-game results (Table 33 in our paper), while the second reports the new length-matched PGG/CPR evaluation averaged over four encodings.

Lineage and comparisonOriginal K (pp, Table 33)Length-matched K (pp)
SmolLM2, SFT − Base−16.2 ± 2.4−21.18 ± 3.49
SmolLM2, Aligned − Base−18.1 ± 2.4−22.61 ± 2.44
Nemotron-Cascade, SFT − Base−4.9 ± 2.9−5.63 ± 8.88
Nemotron-Cascade, Aligned − Base−13.7 ± 3.8−12.25 ± 9.78

All four length-matched mean K contrasts remain negative, across both complete lineages and both post-trained stages. These results show that the average stage-by-history pattern remains observable when initial input length is controlled under the tested game and encoding conditions.

We also tested fully cooperative scripted histories across five complete model lineages. For each checkpoint, we held the game state and historical actions fixed, compared the next decision under HL2 versus HL40, and used neutral archive text to match the shorter prompt's initial token count to the longer prompt. The resulting Base-relative history contrast K averaged −10.83 pp at SFT (negative in 5/5 lineages) and −8.19 pp at Aligned (negative in 4/5 lineages). These results provide additional evidence that post-training amplifies history-related cooperation changes even when initial prompt length is matched. We will clarify this control in the revised experimental description.

AuthorsOpenReview ↗

Response to Reviewer ZEBk (3/3)

(continued)

Specific revision to the Abstract

As suggested, we will revise the Abstract to make the motivation, new research gap, key result, and implication explicit:

Language-model agents increasingly make decisions for different users and organizations, often in repeated interactions where individual actions affect shared outcomes. Prior work identified a memory curse: giving LLM agents more past interaction can reduce cooperation. We investigate how this history sensitivity changes across stages of post-training. Across seven released model lineages and seven social dilemmas, we compare Base, supervised fine-tuned (SFT), and later Aligned checkpoints in 200-round self-play with two or forty visible history rounds. Averaged equally over six binary-action games within each lineage, longer histories reduce cooperation at every SFT and Aligned checkpoint; the decline exceeds Base's in six of seven lineages at both stages. Most of the average amplification is already present at SFT. These results show why evaluating only a final checkpoint or one memory length can miss important aspects of multi-agent behavior, and motivate assessing sustained cooperation jointly with training stage and interaction incentives.

Specific revision to the Introduction

As suggested, we will add the following paragraph after the opening repeated-public-goods example to make the gap and practical motivation explicit:

Existing work shows that longer visible histories can reduce LLM cooperation (Liu et al., 2026), but leaves open how this response changes between pretrained, supervised fine-tuned, and subsequently aligned models. This distinction matters in multi-agent workflows: agents serving different users may each bear a local cost for verification, assistance, or restraint in using shared resources, while the resulting benefits are shared and may depend on later reciprocation. We therefore compare released checkpoints within each model lineage, holding the game rules and evaluation conditions matched, to characterize how training stage interacts with memory length and payoff incentives. This links post-training evaluation to sustained cooperation, rather than isolated decisions alone.

Thank you again for your thoughtful feedback. We hope our new results and clarifications address your concerns, and would greatly appreciate it if you could consider revisiting your rating.

References

  • Tewolde, Emanuel, et al. "Coopeval: Benchmarking cooperation-sustaining mechanisms and llm agents in social dilemmas." Proceedings of the 43rd International Conference on Machine Learning (2026). PMLR 306, 120222–120287
  • Liu, Jiayuan, et al. “The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agents.” arXiv preprint arXiv:2605.08060 (2026).
  • Piatti, Giorgio, et al. "Cooperate or collapse: Emergence of sustainable cooperation in a society of LLM agents." Advances in Neural Information Processing Systems 37 (2024): 111715-111759.
  • Leibo, Joel Z., et al. "Scalable evaluation of multi-agent reinforcement learning with melting pot." International conference on machine learning. PMLR, 2021.
  • Trivedi, Rakshit S., et al. "Melting pot contest: Charting the future of generalized cooperative intelligence." Advances in Neural Information Processing Systems 37 (2024): 16213-16239.
AuthorsOpenReview ↗

Additional Perspective on the Broader Significance of Our Results

We would lastly like to add an additional note that might help with situating our paper in nascent AI advancements:

There has been recent discussions on where in the LLM training pipeline we should infuse safety training. So far, the popular paradigm has focused on safety training as a post-hoc patch once the desired level of LLM capabilities have been achieved (which has been amusingly illustrated as the Shaggoth with a smiley face, cf. this NY Times article https://www.nytimes.com/2023/05/30/technology/shoggoth-meme-ai.html). As this has been found to be fragile, more recent works propose a safety-first pipeline of LLM training, going as far back as to intervene in the early stages of pretraining (Maini et al. NeurIPS 2025; Aydin et al. Communications of the ACM 2026). When it is best to infuse safety training remains an active area of research.

To our understanding, our work is the first to take a step towards that question from the perspective of multi-agent safety, and cooperative capabilities more specifically. With it, our paper connects that line of work to another nascent field of Cooperative AI (Dafoe et al. Nature 2021; Hammond et al. 2025). Importantly, our findings reveal that much of the amplification of the memory curse already emerges during SFT, indicating that cooperative capabilities can be substantially affected well before the final alignment stage. With access to more fine-grained intermediate checkpoints, our methodology can further pinpoint which training stages most strongly affect different aspects of cooperative behavior, helping researchers identify where and how to intervene.

We demonstrate one example of this finer-grained analysis in Appendix T, which examines seven released checkpoints of Nemotron-Cascade: Base, SFT, RLHF, instruction-following RL (IFRL), mathematical RL (MathRL), CodeRL, and software-engineering RL (SWE-RL). Among the transitions between these stages, RLHF → IFRL produces the largest mean decline in payoff responsiveness, whereas SFT → RLHF produces the largest mean decline in history responsiveness. This illustrates how examining intermediate checkpoints can reveal specific stages and behavioral dimensions that warrant targeted investigation.

Identifying which training stages most affect particular cooperative capabilities provides concrete directions for developing effective interventions. Researchers can investigate the training data, objectives, and design choices associated with these stages, explore ways to preserve cooperative capabilities while improving other model capabilities, and evaluate cooperation throughout the training pipeline rather than only at the final checkpoint. We believe that identifying where cooperative behavior changes is an important step toward understanding how to preserve and strengthen it, opening up exciting directions for future research on when and how to intervene most effectively.

References:

Maini et al. 2025. Safety Pretraining: Toward the Next Generation of Safe AI. NeurIPS 2025.
Aydin et al. 2026. From Model Training to Model Raising. Communications of the ACM 2026.
Dafoe et al. 2020. Cooperative AI: machines must learn to find common ground. Nature 2021.
Hammond et al. 2025. Multi-Agent Risks from Advanced AI. 2025.


We hope this additional perspective helps address your concerns about the broader significance, practical relevance, and implications of our work for LLM post-training and Cooperative AI. If so, we would sincerely appreciate your consideration of a higher rating. Thank you again for your thoughtful feedback!

Reviewer_vRC3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper builds on the results of Liu et al. (2026), which established a memory curse for LLM cooperation, asking how the post-training process impacts this curse. The authors perform experiments across a variety of model checkpoints to demonstrate that SFT and Alignment tend to make the memory curse even more pronounced.

Critical strengths
  • In my opinion, the biggest strength of this paper is that it comes to a counterintuitive empirical conclusion. The fact that post-training makes models less likely to cooperate is counterintuitive given the nature of the SFT and alignment phases.
  • The experiments include an impressive coverage of models, checkpoints, and scenarios for the empirical study.
  • The paper contributes to greater understanding of the memory curse of cooperation and what training phases contribute to it most.
Critical weaknesses
  • The paper's focus is restricted to understanding a bit more about the memory curse and stops short of pursuing solutions to the problem.
  • The scope of the contribution is relatively limited in that the memory curse has already been established and the novel insight of this paper is its relation to the phase of post-training. As such, the understanding of the memory curse has been enhanced by this paper, but this topic may be of limited interest. While this is an interesting selection of games for this use case, the fact that the focus is on canonical games leaves a large degree of extrapolation needed for projecting societal impact.
  • The writing quality can be poor at times, bogging down readers in low-level details rather than keeping a coherent high-level narrative structure. This makes sense given the author's claim that they used AI for "result interpretation, literature synthesis, manuscript drafting, and figure/table preparation." In particular, it is glaring that AI was used for generating the figure captions.
Questions for authors
  • Q1: I would be interest to here the take of the authors on what progress this paper has made or not made towards helping algorithm developers address this memory curse.
  • Q2: For what real-world use cases should readers be most concerned about lack of cooperation and how do these setups relate to the configuration of the 7 games considered?
6 comments
AuthorsOpenReview ↗

Response to Reviewer vRC3 (1/4)

We thank the reviewer for highlighting our paper’s counterintuitive finding that post-training can deepen the decline in cooperation as more history becomes visible, and for recognizing the range of models and settings in the evaluation.

Q1 and W1/W2a · Progress toward addressing the memory curse and implications for agentic post-training

Thank you for raising this important question. The central contribution of CoopShift is not merely documenting another instance of the memory curse, but identifying how the post-training pipeline affects cooperation in repeated interactions and what this implies for developing more cooperative agents. We believe this contribution is of broad interest to the multiagent safety and AI x Game Theory community.

(I). Identifying where the problem emerges during post-training. Our results (Table 3 in our paper) show that the mean cooperation change from two to forty visible rounds is −1.17 pp at Base, −8.11 pp at SFT, and −10.93 pp at Aligned, averaged across seven model lineages and six binary-action social dilemmas. Most of the average amplification therefore already emerges at SFT, rather than only during subsequent RL or alignment phases. This highlights the importance of evaluating intermediate as well as final checkpoints when developing agents for sustained interaction.

(II). Evaluation principles for future mitigation methods. Our results motivate two concrete criteria for evaluating potential mitigations: (i) whether they help sustain beneficial cooperation as interaction histories grow, and (ii) whether the resulting decisions remain appropriate under different incentives and opportunities for reciprocity. The first follows from the stage-by-history effects in Table 3 of our paper. The second is motivated by our payoff-response analysis (Table 2) and controlled partner-policy study (Appendix U), which show why the value of contributing depends on the payoff structure and how partners respond. Together, these findings provide a basis for evaluating mitigation methods across both history lengths and strategic conditions.

Our new Always-Cooperative Partner Control identifies an additional condition worth testing. We examined whether the history-related decline requires uncooperative partners. In three-player repeated social dilemmas, one player was controlled by the evaluated LLM checkpoint, while the other two partners cooperated throughout the interaction, including the initial 40-round warm-up. The evaluated LLM continued making its own decisions. We compared two versus forty visible history rounds (HL2 vs. HL40), measuring cooperation over rounds 47–100. The experiment covered SmolLM2 and DeepSeekMath at Base, SFT, and Aligned stages, six payoff configurations across PGG, CPR, and Donation-PD, and ten new sampling seeds.

The table reports $H=\mathrm{Coop}(HL40)-\mathrm{Coop}(HL2)$, equally averaged across the six payoff configurations (mean ± SD across ten seeds, in pp).

CheckpointHL40 − HL2 cooperation (pp)
SmolLM2 Base+0.11 ± 0.44
SmolLM2 SFT−19.51 ± 8.10
SmolLM2 Aligned−20.23 ± 5.30
DeepSeekMath Base+0.34 ± 1.46
DeepSeekMath SFT−17.82 ± 4.18
DeepSeekMath Aligned−15.62 ± 1.62

Across both lineages, the Base checkpoints show near-zero mean changes, while all four post-trained checkpoints exhibit substantial negative mean changes under longer histories, even with consistently cooperative partners. This identifies another important evaluation condition for future mitigation methods: history sensitivity should also be tested with reliable partners, rather than focusing exclusively on responses to partner defection.

AuthorsOpenReview ↗

Response to Reviewer vRC3 (2/4)

(continued)

(III). Implications for agentic post-training and multi-agent incentives. These findings suggest that agentic post-training should prepare models not only to complete individual tasks, but also to make effective decisions when their actions influence other agents and shared outcomes. The objective is to support cooperation that accounts for the agent's own incentives, its partners' responses, and the benefits of collaboration.

To illustrate the practical significance of our findings, we additionally analyzed realized payoffs in the new Always-Cooperative Partner Control. In the three-player Public Goods Game with multiplier (r=1.2), we compared DeepSeekMath SFT under HL2 and HL40 while keeping both partner policies fixed. We measured the evaluated player's individual payoff and the total payoff of all three players, averaged per round.

Payoff metric (points per round)HL2HL40
Evaluated agent's individual payoff1.5791.712
Total group payoff3.4743.429

With longer visible history, the evaluated agent receives a higher individual payoff while the group's total payoff is lower. This illustrates how history-dependent cooperation decisions can affect individual and collective outcomes differently, highlighting the importance of incentive-aware post-training for multi-agent systems with partially aligned objectives.

(IV). A potential direction for mitigation. Our findings motivate exploring whether collaboration-focused or incentive-aware post-training may benefit from starting at an intermediate checkpoint (e.g., SFT), rather than the final Aligned model. In future work, we plan to investigate this through larger-scale, compute-matched post-training experiments targeting cooperative and incentive-sensitive agent behaviors, evaluating whether the choice of initialization model checkpoint improves sustained cooperation across different partner strategies and application domains.

We will expand the Discussion to make these evaluation principles and future research directions explicit.

Q2 and W2 · Real-world applications and the seven incentive structures

The growing use of AI agents representing different users and organizations makes cooperation under partially aligned incentives an increasingly important practical problem. Agents may pursue different objectives while still depending on one another to achieve mutually beneficial outcomes. This is already emerging in agent-mediated commerce. CarEdge offers a consumer-facing AI Buying Agent (https://cs2.caredge.com/live) and dealership-facing AI products (https://caredge.com/for-dealers), and Mark Miller Subaru (https://www.markmillersubarumidtown.com/blog/ai-agent-car-sale) has reported an agent-to-agent vehicle transaction involving a CarEdge buyer agent and a dealership-built agent. Luminance (https://www.luminance.com/press-releases/luminance-showcases-worlds-first-completely-ai-powered-contract-negotiation) has demonstrated AI agents negotiating a contract on behalf of opposing parties, while Pactum (https://pactum.com/campaigns-agents) develops agents that negotiate commercial terms with suppliers.

These interactions create opportunities for cooperation even when the agents' objectives differ. For example, buyer and seller agents may disagree on price, but both benefit when transaction details are accurate and errors are resolved efficiently. One agent might spend additional time checking quoted fees, correcting transaction information, or resolving a delivery issue. That effort consumes its own resources while creating benefits for both parties, introducing a strategic choice between contributing and relying on the other party's effort. In recurring supplier relationships, such choices can also influence future assistance and reciprocity. Reputation adds another dimension: CarEdge's Dealer Transparency Index (https://caredge.com/rating-methodology) uses historical quotes to evaluate pricing transparency, illustrating how earlier interactions can inform future counterparties. CoopEval similarly studies how repeated interaction and reputation can support cooperation among LLM agents (Tewolde et al., 2026).

AuthorsOpenReview ↗

Response to Reviewer vRC3 (3/4)

(continued)

The seven games in CoopShift provide controlled settings for studying distinct incentive structures that arise in these applications:

GameCorresponding real-world strategic decision
Public Goods GameWhether to contribute time or resources to shared verification, documentation, or testing that benefits multiple agents
Common-Pool ResourceWhether to limit one's use of shared compute, API capacity, or other constrained resources for the benefit of the group
Donation Prisoner's DilemmaWhether to spend effort helping another agent when assistance may be reciprocated in future interactions
Volunteer's DilemmaWhether to take responsibility for an incident or alert when one responder is sufficient to benefit everyone
Assurance GameWhether to participate in a shared migration or deployment whose benefits depend on enough agents joining
Traveler's DilemmaHow agents choose claims when the payoff rules explicitly reward lower claims and penalize higher claims
Trust GameWhether to entrust resources to another participant and how much to return, where our implementation elicits both roles' intentions simultaneously

Together, these games allow us to systematically examine how agents respond to different costs, shared benefits, reciprocity, coordination requirements, and resource constraints. By varying the payoff structure while studying the same training-stage comparisons, CoopShift helps characterize how post-training changes cooperation across different strategic environments.

This research direction is also motivated by related work. CoopEval examines cooperation-sustaining mechanisms, including repeated interaction, reputation, mediation, and contracts (Tewolde et al., 2026). GovSim studies cooperative management of shared resources over time (Piatti et al., 2024). Melting Pot and its NeurIPS 2024 contest investigate cooperation and generalization to unfamiliar partners in richer multi-agent environments (Leibo et al., 2021; Trivedi et al., 2024).

CoopShift adds a complementary finding: post-training itself can substantially change how LLM agents respond to accumulated interaction history in these strategic settings. This connects our empirical results to an important challenge in building reliable multi-agent systems: evaluating not only what individual agents can accomplish, but also how their cooperation evolves across training stages, incentives, and repeated interactions. We will strengthen the Introduction and Discussion with these concrete applications, the game-to-decision mappings above, and their implications for agentic post-training and evaluation.

W3 · Strengthening the paper's narrative and presentation

Thank you for the specific feedback on presentation. We will make the paper's central contribution and supporting evidence easier to follow, with a clearer high-level narrative throughout. In the Introduction, we will present the main analysis in two steps: first, measure how cooperation changes when visible history increases from two to forty rounds within each checkpoint; second, compare this history-related change across Base, SFT, and Aligned checkpoints within the same model lineage. This will make the key stage-by-history finding explicit before introducing detailed comparisons.

We will also strengthen the organization and presentation of the results. Section 3 examines how cooperation varies with payoff incentives; Section 4 and Table 3 establish the main stage-by-history finding; and Section 5 examines how cooperation differs across interaction settings and expectations of future play. Figure 2 provides the corresponding breakdowns across games, payoffs, and model lineages. We will add concise takeaways to these sections, improve transitions, and keep implementation-specific details in the methodological sections and appendices. As suggested, we will also revise figure captions, including Figure 1, to clearly explain the experimental settings and plotted quantities.

Comments on the paper

1 comment

My concerns have been addressed. No further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.