acceptodds
Under review as a conference paper at ICLR 2027

OppoMem: An Opponent-Aware Memory Framework for LLM Agents Evolving in Strategic Games

Abstract

Memory enables large language model (LLM) agents to evolve through experience without updating model parameters. However, strategic games pose three key challenges: uncertainty arising from opponent behavior, the limitations of naive similarity-based memory retrieval, and difficulty evaluating individual memories from final outcome feedback. Inspired by opponent modeling in game theory, We propose **OppoMem**: an **Oppo**nent-aware **Mem**ory framework for self-evolving LLM agents in strategic games. OppoMem represents strategic memories as associations between opponent types and response strategies. The agent uses online opponent modeling to guide the dynamic retrieval of relevant memories. To support memory evolution, OppoMem estimates individual memory utility through randomized activation and uses the resulting feedback to guide memory retention, revision, and removal. Extensive experiments show the effectiveness of our framework in enabling LLM agents to evolve in strategic games with an average relative performance improvement around 30%. Our code is available at https://anonymous.4open.science/r/Core_code-91FF/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Memory-Augmented LLM Agents;Strategic Games;Opponent Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MHQ6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents OppoMem: an Opponent-aware Memory framework for self-evolving LLM agents in strategic games. Oppomem utilizes an opponent-conditioned retrieval mechanism that dynamically selects memories based on online opponent modeling, together with a more effective memory evaluation mechanism that estimates the utility of individual memories in long-horizon decision-making scenarios. The paper presents extensive experiments across diverse game settings demonstrate the effectiveness of OppoMem and establish opponent modeling and memory utility evaluation as key components for building effective self-evolving agents.

Critical strengths

The paper presents a novel strategy where the memory is indexed by opponent type and retrieved by the agent first identifying the opponent type from observation.

Critical weaknesses

The paper did not present qualitative analysis and studies on variants of the setting, which could potentially lead to more interesting discoveries and applications (see questions).

Questions for authors
  1. Since the label is derived from the full trajectory while the agent determination is derived from oartial observations, does the author observe impact or failure mode at the earlier stage of the game when the data is less sufficient
  2. Are the range of opponents types modeled or studied? Can different LLMs have blund spot on particular opponents model? Can the model be given a prompt to introduce it to more exotic opponents models.
  3. How can the current setup extend to the consideration of the opponent agent using opponent models on the agent itself? What about nore levels of awareness.
  4. Can the framework be appointed to analyze the recent openai incident on huggingface? Eg a prisoner dillema between agent choosing to report to human the group jail break activity versus following the group.
7 comments
AuthorsOpenReview ↗

Response to Reviewer MHQ6 (1/4)

Hi Reviewer MHQ6,

Thank you very much for your valuable time and detailed suggestion. We agree that examining OppoMem from qualitative analysis and studies could lead to more interesting discoveries and applications. In the rebuttal below, we provide additional experiments and analyses to address the questions you raised.

Early-Stage Opponent Modeling

Q1: Since the label is derived from the full trajectory while the agent determination is derived from oartial observations, does the author observe impact or failure mode at the earlier stage of the game when the data is less sufficient.

TL;DR: We focus on final outcomes in long-horizon games, with early uncertainty addressed through continual opponent-model updates.

Action: We will include an analysis of this issue in the experiment section.

Thank you again for taking a close look at our paper. Indeed, opponent type is a very important part of our method. Your question is indeed reasonable: in the early stage of a game, the agent may model the opponent inaccurately because it has only a partial observation history.

First, most of the games we focus on are long-horizon tasks, such as RPS and IPD in the environments we use, and our main optimization objective is the final win or loss of each game. The use of opponent modeling by LLM-based agents in these kinds of scenarios has been shown to significantly improve game ability, such as in Hypothetical minds [1].

Our ablation experiments also show that explicitly asking the agent in the prompt to model the opponent before making a decision can improve the agent's game ability; see the “w/o memory” row in Table 2.

Second, we also paid attention to the issue you mentioned when designing our method. So, before each decision, we let the agent explicitly update the previous round's opponent model based on the latest observation, to make it as accurate as possible, while also allowing it to output that it cannot determine the opponent model when the number of samples is insufficient.

We found a case in RPS against a rule-based bot to show the issue you mentioned. The opponent's strategy is to counter the move that our side has played most frequently in the historical trajectory. The changes in our guesses about the opponent's strategy over 20 rounds are as follows:

R1: Insufficient evidence.
R2: Insufficient evidence.
R3: May counter our previous move.
R4: Counters our previous move.
⋮
R11: Counters our previous move, except after two consecutive Scissors from us.
R12: Revises the hypothesis: counters our most frequent historical move.
⋮
R20: Correctly predicts Paper from historical counts (R/P/S = 7/6/6), despite our last move being Scissors. The bot indeed plays Paper.

This trajectory shows that the agent initially struggles to identify the correct opponent type due to limited interaction history. As the history grows, counterexamples prompt the agent to revise its hypotheses until it identifies the underlying mechanism of the opponent’s strategy. Further observations help confirm and refine this understanding, which then guides the agent’s decisions and memory retrieavl.

[1] Cross L, Xiang V, Bhatia A, et al. Hypothetical minds: Scaffolding theory of mind for multi-agent tasks with large language models[C]//International Conference on Learning Representations. 2025, 2025: 6507-6546.

AuthorsOpenReview ↗

Response to Reviewer MHQ6 (2/4)

Scope of Opponent Types

Q2.1: Are the range of opponents types modeled or studied?

TL;DR: Opponent types are inferred from trajectories and clustered for memory retrieval, without a predefined taxonomy.

Action: We will include an analysis of this issue in the Appendix section.

Thank you again for your thorough observation and interesting question. First, please allow us to repeat our definition of opponent type. As you mentioned in question 1, opponent type is obtained from the complete trajectories collected, rather than being predefined.

Since different opponents may have the same behavior pattern, the same opponent may exhibit different behavior patterns at different stages, and the ego agent also explicitly models the opponent's behavior pattern semantically from a partial observation history, opponent types are very numerous and complex. It is precisely for these reasons that we cluster these opponent types and store them in embedding form for subsequent use in memory retrieval.

We do not assume a predefined, closed taxonomy of opponent types. On the one hand, the agents we test are LLM-based and usually show different behavior patterns in different games, and even show different behavior patterns in the same game, so opponent type has no ground truth. On the other hand, as in our response to your question 1, our method is more focused on how to help the ego agent make decisions through better memory retrieval, rather than on better obtaining opponent types. We present 8 representative opponent types summarized from RPS as a case study:

Representative patternFixed actionBiased mixturePeriodic cycleNon-repetitionLag-one imitationLag-one counteringHistorical-frequency exploitationOutcome-conditioned response
Description (paraphrased)Always plays Rock, regardless of history.Mostly plays Paper, with occasional Scissors and rare Rock; whether deviations are patterned or random is unclear.Follows a fixed Rock → Paper → Scissors cycle.Never repeats its own previous move.Consistently copies ego's previous move in the observed rounds, with no other clear pattern.Tends to counter ego's previous move, especially in rounds 8–19, with earlier exceptions.Counters ego's most frequent past move; ties favor Scissors, then Paper, then Rock.Repeats its winning move in the next round.
AuthorsOpenReview ↗

Response to Reviewer MHQ6 (3/4)

Model-Specific Blind Spots

Q2.2: Can different LLMs have blund spot on particular opponents model?

TL;DR: Different LLMs do indeed have blind spots when identifying opponent types.

Action: We will add this analysis to the Appendix.

Very interesting question. For the RPS setting, we used 10 types of rule-based bots as opponents, tested each bot for 40 games, and each game had 20 rounds. We asked three LLMs to guess the opponent's type at rounds 10 and 20. We did not use predefined labels; instead, we asked the LLMs to describe it in natural language, and then used GPT-6.1-Sol to judge whether the guess was correct. We report the success rates at rounds 10 and 20 as follows:

Success rates at round 10:

rule-based botClaude Sonnet 5Deepseek-v4.1-flashGPT-5.6-Terra
Always plays the same move100% (40/40)100% (40/40)100% (40/40)
Favors one move (70%)45% (18/40)38% (15/40)75% (30/40)
Chooses R/P/S uniformly at random10% (4/40)5% (2/40)43% (17/40)
Cycles through R → P → S100% (40/40)100% (40/40)100% (40/40)
Copies ego's last move93% (37/40)100% (40/40)95% (38/40)
Counters ego's last move100% (40/40)100% (40/40)90% (36/40)
Counters ego's move from two rounds ago5% (2/40)45% (18/40)0% (0/40)
Counters ego's most frequent move in the last three rounds0% (0/40)15% (6/40)0% (0/40)
Repeats after a win; cycles forward after a loss or draw68% (27/40)73% (29/40)25% (10/40)
Copies ego's last move in rounds 1–10; counters it in rounds 11–20---

Success rates at round 20:

rule-based botClaude Sonnet 5Deepseek-v4.1-flashGPT-5.6-Terra
Always plays the same move100% (40/40)100% (40/40)100% (40/40)
Favors one move (70%)53% (21/40)60% (24/40)83% (33/40)
Chooses R/P/S uniformly at random20% (8/40)0% (0/40)55% (22/40)
Cycles through R → P → S100% (40/40)100% (40/40)100% (40/40)
Copies ego's last move100% (40/40)100% (40/40)100% (40/40)
Counters ego's last move100% (40/40)100% (40/40)100% (40/40)
Counters ego's move from two rounds ago13% (5/40)88% (35/40)5% (2/40)
Counters ego's most frequent move in the last three rounds0% (0/40)20% (8/40)0% (0/40)
Repeats after a win; cycles forward after a loss or draw93% (37/40)100% (40/40)45% (18/40)
Copies ego's last move in rounds 1–10; counters it in rounds 11–20100% (40/40)98% (39/40)75% (30/40)

Overall, from these two tables, LLMs generally had a higher success rate in guessing at round 20 than at round 10. As you said in question 1, more trajectories allow LLMs to judge the strategy of a particular opponent more accurately.

Regarding your concern about whether different LLMs have blind spots, we also found some interesting conclusions. For example, Deepseek-v4.1-flash has difficulty guessing the correct strategy of an opponent that makes purely random moves, but instead performs well on a case such as “Counters ego's move from two rounds ago,” while GPT-type and claude-type models perform poorly. Similarly, we also found that for a relatively implicit strategy such as “Counters ego's most frequent move in the last three rounds,” all three models performed poorly. For more obvious strategies such as “Copies ego's last move” and “Counters ego's last move,” all three models performed well.

Prompting for More Exotic Opponents

Q2.3: Can the model be given a prompt to introduce it to more exotic opponents models.

TL;DR: Targeted prompts can improve the identification of the specific opponent strategies.

Action: We will add this analysis to the Appendix.

We conducted two types of prompt injection experiments. First, for the case where GPT-type and claude-type models did poorly at considering historical trajectories, we added “You need to consider a longer history” to the prompt and reran the experiment. Here, we report only the round-20 guessing results, as follows:

rule-based botClaude Sonnet 5GPT-5.6-Terra
Counters ego's move from two rounds ago47% (19/40)40% (16/40)
Counters ego's most frequent move in the last three rounds5% (2/40)2% (1/40)

We found that compared with the original prompt, targeted prompt injection did indeed bring a noticeable improvement.

Second, for Deepseek-v4.1-flash's inaccurate guesses for purely random moves, we added “You need to distinguish randomness from game strategies” to the prompt. The round-20 guessing result of Deepseek-v4.1-flash was still 0%. We tried several similar prompts; except when explicitly telling it that “the opponent's strategy is to make random moves,” Deepseek-v4.1-flash could not guess it in the other cases.

So from the above experiments, we can see that different models indeed perform differently when facing opponents with different strategies, and specific prompts can improve different models' ability to recognize opponent strategies.

Reviewer_AUbGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OppoMem, an opponent-aware memory framework for LLM agents in strategic environments. The framework revolves around a memory bank whose entries are key-value pairs. The keys are opponent types, and the values the corresponding strategies to follow against each type. The memory bank is built by running games against a heterogeneous pool of opponents (rule-based, and LLM-based), with each decision step contributing a tuple of history, inferred opponent type (natural language), action and final game outcome. The collected opponent descriptions are clustered, and a reflection LLM (GPT-5.6 Sol) is used to generate response strategies for each cluster. The memory bank undergoes several rounds of refinement. These involve simulating games with each memory being randomly turned on or off in order to estimate its utility during game play. Low-utility memories are then deleted or revised by the LLM, and new opponent types are added to the bank if present. At inference, using the final, frozen memory bank, the agent infers the opponent type, matches the type with the bank’s keys, and receives the corresponding strategy in its context to generate the next action.

The authors evaluate OppoMem in 4 environments (IPD, Repeated RPS, Hanabi, and Deliberation) with two distinct backbones (DeepSeek-V4-Flash, GPT-5-mini) against GPT-5.5-based opponents (except Hanabi, which uses self-play). They compare their method against 7 baselines (ReAct, Reflexion, ExpeL, A-MEM, Agent-Pro, Strategist, and the Vanilla LLM), and claim an average relative improvement of 30%. Additionally, they conduct three ablations: a memory bank with history summaries (rather than opponent types) as keys, no dynamic memory retrieval (full memory bank fed to the LLM agent), and no memory bank (but retaining online opponent modelling).

Critical strengths
  • The authors compare their method against six relevant baselines across both backbones, including generic and game-theoretic specific self-improvement methods.
  • The authors employ a principled memory utility estimation method that relies on randomised memory activation during training. For each game, the first time each memory is retrieved, it is randomly turned on or off, and remains in the same state for the remainder of the game. After collecting multiple game trajectories, the memory’s utility is estimated by subtracting the mean final outcome of the games where the memory was absent from the one where it was present. This alleviates the memory credit assignment problem that arises from multi-step interactions.
Critical weaknesses
  • There are several unsupported claims throughout the introduction and related work section, such as ExpeL and A-MEM having poor robustness to uncertainty (lines 52-53), and Agent-Pro not being adequate for strategic interactions due to the inherent uncertainty of strategic games (lines 116-118, despite the original paper evaluating the agent on imperfect information games). These claims are employed to motivate and position the proposed framework. While the authors choose appropriate baselines for the main results in Table 1, the conceptual motivation and positioning of their work with respect to the relevant literature (Agent-Pro, Strategist, Deep BPR+) is lacking in the introduction and related work section.
  • One of the central benefits of the method is claimed to be robustness to the uncertainty brought by opponents’ behavioural changes (92-93). However, the main results in Table 1 only report performance against an unseen, single opponent model (GPT-5.5, except for Hanabi, which uses self-play). While this partially tests generalisation, it does not support better robustness against a pool of heterogeneous opponents or opponents that modify their strategy mid-play. Additionally, OppoMem makes use of a reflection LLM (GPT-5.6 Sol) to generate the response strategies to the opponent types. This may constitute a capability confound, since GPT-5.6 Sol is much more capable than both backbones.
Questions for authors
  • Could the authors adjust the introduction and related works sections, removing unsupported claims and positioning their framework more thoroughly against prior work, especially Agent-Pro and Deep BPR+?
  • Could the authors confirm whether GPT-5.6 Sol was used in any of the baselines (and if so, for which component specifically), and run an ablation where the reflection LLM in OppoMem is replaced by the backbone LLM (i.e. using DeepSeek-V4-Flash and GPT-5-mini)?
  • Could the authors evaluate OppoMem and the corresponding baselines against more opponents that were unseen during training (both LLM agents, and rule-based opponents, as well as opponents that change their strategy mid-game)?
13 comments
AuthorsOpenReview ↗

Response to Reviewer AUbG (1/4)

Hi Reviewer AUbG,

Thank you very much for your careful reading and constructive suggestions. We appreciate this opportunity to clarify our motivation and better position OppoMem relative to prior work. Below, we provide a more detailed conceptual comparison, supported by our experimental results, to address your concerns.

Clarifying the Motivation and Positioning of OppoMem

Q1: Could the authors revise the Introduction and Related Work to remove unsupported claims and provide a more comprehensive positioning of their framework relative to prior work, particularly Agent-Pro and Deep BPR+?

TL;DR: We clarify the motivation and evidence underlying our original claims, particularly regarding opponent-induced uncertainty and experience reuse in strategic games. We also provide a more comprehensive comparison with prior work, to clarify OppoMem's distinct research focus and contributions.

Action: We will revise the Introduction and Related Work to provide a more comprehensive positioning of OppoMem within the existing literature.

Thank you again for taking a close look at our paper. We agree that the motivation behind our claims and OppoMem’s positioning relative to prior work should be explained more clearly and comprehensively. Below, we provide a more detailed comparison with the relevant literature, particularly Agent-Pro and Deep BPR+.

Positioning Relative to ExpeL and A-MEM

ExpeL and A-MEM are memory frameworks designed for conventional tasks such as long-horizon memory. ExpeL distills general insights from past successes and failures, while A-MEM organizes experiences into evolving, semantically linked notes. Both perform retrieval via similarity over situational information including the current task and state.

In strategic interactions, however, the effectiveness of a response hinges on the opponent’s Opponent Type, since similar observed interaction histories do not necessarily entail that a previously successful strategy remains applicable. As a concrete example, in the IPD, an opponent cooperate for the first N−1N-1 rounds and then defect at round NN. The history remains predominantly cooperative, but the latest defection change which response is appropriate. Retrieval based on overall history similarity favors cooperative experiences, even though the behavioral assumptions underlying their success may no longer hold. OppoMem instead uses the new observation to update its hypothesis about the opponent’s behavior and retrieve responses conditioned on that hypothesis, including a defensive response at round N+1N+1.

OppoMem explicitly grounds the construction and retrieval of strategic memory on inferred opponent types. At each decision step, OppoMem updates its hypotheses over opponent types based on the observable interaction history and uses these hypotheses to retrieve responses suited to the opponent’s inferred behavior.

The ablation results in Table 2 provide empirical support for this opponent-conditioned memory design:

GameOppoMem (Full)w/o Opponent ConditioningScore Difference
RPS64.355.3+9.0
IPD52.550.8+1.7
Hanabi55.246.8+8.4
Deliberation66.858.4+8.4

The ablation replaces opponent-conditioned memory construction and retrieval with memories distilled from complete trajectories and retrieved based on game-history similarity.

The performance reductions across all four games suggest that conditioning memory construction and retrieval on opponent behavior provides benefits over history-similarity-based experience reuse.

Positioning Relative to Agent-Pro

Agent-Pro incorporates dynamic belief modeling, policy-level reflection, and DFS-based policy optimization, demonstrating the effectiveness of strategic self-improvement in imperfect-information games. Our discussion of opponent-induced uncertainty concerns a specific generalization challenge: whether strategic knowledge acquired against one opponent pool remains effective against previously unseen opponents.

Agent-Pro's original evaluation uses the same opponent agents across training, development, and testing. In Limit Texas Hold'em, Agent-Pro interacts with three opponent agents (DQN, DMC, and GPT-3.5). The training, development, and test sets differ in sampled game hands and player positions, while the opponent agents remain the same. The development set is also used for DFS-based policy optimization and selection.

Thus, while Agent-Pro demonstrates effective strategic improvement and dynamic belief adaptation within its evaluated settings, its original experiments do not examine generalization to held-out opponent agents. This leaves open the question of how well its evolved policies generalize when facing opponents with different behavioral patterns.

(The remainder of our response is provided on the next page.)

AuthorsOpenReview ↗

Response to Reviewer AUbG (2/4)

To investigate this question, OppoMem explicitly evaluates experience reuse against held-out LLM opponents. Both Table 1 and the additional evaluation in Q3 compare Agent-Pro and OppoMem against opponents excluded from their respective training pools. Q3 further extends the evaluation to held-out rule-based variants, opponents that switch policies mid-game, and additional LLM opponents. In Table 1 and the additional evaluation in Q3, OppoMem achieves higher mean scores than Agent-Pro.

Positioning Relative to Strategist

Strategist introduces a bi-level framework for strategic self-improvement, using self-play and reflection to optimize high-level strategies, such as value heuristics and dialogue guides. These strategies are evaluated through self-play and improved to achieve better overall performance. For game-action selection, Strategist uses decision-time MCTS with repeated simulated rollouts guided by learned value heuristics, rather than directly prompting an LLM to output the action.

The key difference is that Strategist mainly learns general strategic guidance through self-play, whereas OppoMem accumulates and reuses experience conditioned on different opponent behaviors. Rather than optimizing a general strategy across simulated interactions, OppoMem organizes experience from different training opponents into opponent-type–response-strategy memories. During interaction, OppoMem continuously identifies the opponent's current behavior and retrieves the corresponding memory. This allows the agent to reuse strategies learned from different opponents and adapt its responses as their behavior changes.

Another difference lies in how these strategies are used. Strategist employs MCTS-based planning with simulated trajectories, whereas OppoMem directly retrieves accumulated strategic memories without requiring decision-time MCTS rollouts. This provides an alternative when forward simulation is costly or difficult.

Positioning Relative to Deep BPR+

Deep BPR+ combines opponent modeling with response-policy reuse against non-stationary opponents. OppoMem investigates how opponent modeling can organize experience across episodes into retrievable and maintainable natural-language memories, enabling LLM agents to efficiently accumulate and reuse strategic experience. The two frameworks differ in application settings, adaptation mechanisms, and experience representation and evolution.

First, the two methods operate in fundamentally different settings: Deep BPR+ is a deep reinforcement learning framework based on neural policies, whereas OppoMem is a memory-augmented LLM agent. Deep BPR+ is evaluated in three structured Markov games. Its opponent-policy detection experiments involve 6, 5, and 5 predefined policies, respectively, with separate experiments examining adaptation to unseen policies. In contrast, OppoMem studies experience accumulation and reuse in LLM-based strategic games that cover competitive, cooperative, and mixed-motive interactions, allowing us to examine opponent-aware experience reuse across more varied interaction settings.

Second, the frameworks differ in how they reuse experience and adapt to new opponents. Deep BPR+ uses Bayesian opponent modeling to identify opponent policies. When encountering unfamiliar strategies, it can initialize a new response policy through its Distilled Policy Network (DPN) and further optimize it through online reinforcement learning. Thus, when existing policies are insufficient, adaptation still requires additional online interaction and parameter optimization. OppoMem instead infers opponent behavior from observable interaction history and retrieves a matching natural-language response strategy. In our RPS, IPD, and Deliberation evaluations, the memory bank remains frozen during testing against held-out LLM opponents, examining whether accumulated strategic experience can directly support decisions.

**Third, the frameworks use different units for accumulating and maintaining experience across episodes. ** Deep BPR+ primarily encodes experience in the parameters of neural response policies. Consequently, experience accumulation primarily occurs at the parameterized-policy level, without directly supporting fine-grained maintenance of individual experiences and their applicability conditions; expanding its response capabilities may also require further training.

In contrast, OppoMem explicitly stores opponent-behavior–response-strategy associations as natural-language memories, using their applicability conditions to guide organization and retrieval. This design enables strategic experience to accumulate across episodes as independently editable units and to be retrieved as needed in subsequent interactions, without reoptimizing response-policy parameters for each experience update.

AuthorsOpenReview ↗

Response to Reviewer AUbG (3/4)

Impact of Reflection LLM Capability

Q2: Could the authors clarify whether GPT-5.6-sol was used in any baselines (and, if so, for which components), and provide an ablation study replacing OppoMem's reflection LLM with the backbone LLMs (DeepSeek-V4-Flash and GPT-5-mini)?

TL;DR: GPT-5.6-sol was used for offline reflection in five reflection-based baselines to ensure a consistent comparison. Additional ablations show that OppoMem maintains comparable performance when the reflection LLM is replaced with the backbone model itself.

Action: We will clarify the reflection LLM configurations of all baselines in the experimental setup and include the reflection-model ablation results in the experiment section.

Thank you for raising this important concern. We understand that using a stronger reflection LLM may introduce a capability confound, potentially making it difficult to distinguish the contribution of OppoMem's framework from that of the reflection model.

Reflection LLM Usage in Baselines

First, we confirm that GPT-5.6-sol was also used in all five baselines containing reflection or experience-refinement components, rather than being exclusively available to OppoMem.

Specifically, these baselines use trajectories and feedback generated by the backbone model, while GPT-5.6-sol performs the corresponding offline reflection operations:

  • Reflexion: Generates lessons from completed trajectories and feedback.
  • ExpeL: Distills reusable insights and rules from past experiences.
  • A-MEM: Creates and updates memory notes.
  • Agent-Pro: Reflects on trajectories and revises policy prompts.
  • Strategist: Refines high-level strategies based on game feedback.

For all methods( except strategist), DeepSeek-V4-Flash or GPT-5-mini remains the backbone model responsible for action generation. GPT-5.6-sol is used only for offline reflection and refinement, not for action selection.

Our motivation for adopting a stronger reflection LLM was to provide reflection-based methods with high-quality experience refinement, allowing their respective framework designs to be evaluated without being unnecessarily limited by weak reflection capabilities.

Ablation Study: Impact of Reflection LLM Capability

To directly address the reviewer's concern, we conducted additional experiments with different reflection LLMs while keeping the backbone model fixed.

We compare three reflection configurations: GPT-5.6-sol (stronger model), the backbone LLM itself, and GPT-4o-mini (weaker model). Only the reflection LLM used for offline memory distillation and refinement is replaced; all other components remain unchanged.

The experiments follow the same settings and evaluation protocol as the original paper. Results are reported as mean over three random seeds, with higher values indicating better performance.

BackboneReflection LLMRPS ↑IPD ↑Hanabi ↑Deliberation ↑
DeepSeek-V4-FlashGPT-5.6-sol64.753.156.365.9
DeepSeek-V4-Flash (same as backbone)62.751.857.466.5
GPT-4o-mini64.052.952.665.3
GPT-5-miniGPT-5.6-sol47.545.432.042.3
GPT-5-mini (same as backbone)49.044.130.941.6
GPT-4o-mini48.343.928.639.1

Analysis

The results provide two important observations.

First, replacing GPT-5.6-sol with the backbone LLM itself does not lead to a substantial performance degradation. For DeepSeek-V4-Flash, performance remains comparable across all four games, with even slightly higher scores in Hanabi and Deliberation. Similar observations hold for GPT-5-mini, where using the backbone itself achieves comparable results.

Second, a stronger reflection LLM does not consistently produce better performance. Although GPT-5.6-sol achieves higher scores in some settings, the backbone models and GPT-4o-mini also outperform it in several cases. This suggests that reflection model capability is not the sole determinant of OppoMem's effectiveness.

Overall, these results indicate that OppoMem's effectiveness does not critically depend on using a stronger external reflection LLM. The framework remains effective when reflection is performed by the backbone model itself, supporting the contribution of its opponent-conditioned memory organization, dynamic retrieval, and memory evolution mechanisms.

Reviewer_R5ivOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a memory framework named OppoMem for self-evolving LLM agents in strategic games. It stores memories as pairs of opponent-type descriptions and corresponding response strategies. Then it retrieves them dynamically via online opponent modeling. The experiment shows that Oppomem improves performance across many games, outperforming previously commonly used memory frameworks.

Critical strengths
  1. This paper proposes a method to target and solve all 3 challenges when LLM agents evolve in strategic games. These challenges are real and difficult and remain unsolved by previous methods. And the proposed unified memory framework addresses all of them.

  2. The comparison is detailed across multiple well-established, commonly used memory methods and across multiple games. It shows the method’s effectiveness and generalization across different settings.

Critical weaknesses
  1. The paper relies on the LLM to generate opponent-type descriptions, which can be unstable and lack corresponding evaluation. The paper does not evaluate stability or quantify the generation quality. Incorrect descriptions will cause problems for downstream retrieval.

  2. The cross-game transfer study is small-scale and limited to one game. Adding more transfer experiments would better support the generality claims.

  3. The paper lacks a comparison of token consumption across different methods, making it difficult to determine whether OppoMem’s performance gains justify the additional cost of opponent modeling and memory refinement.

  4. Several typos need to be addressed. (See questions.)

Questions for authors

Please see the questions in the weaknesses. What's more, there are several typos in the paper, e.g., “game theory, We propose,” “evaluating an updating memories,” “denfensive,” “no special quote,” “disrbution,” “prefernce,” and “againt.”

4 comments
AuthorsOpenReview ↗

Response to Reviewer R5iv (1/3)

Hi Reviewer R5iv,

Thank you very much for your careful reading and constructive suggestions. We appreciate this opportunity to clarify several aspects of OppoMem and further strengthen our experimental evaluation. We provide additional experiments and analyses to address your concerns regarding opponent-type reliability, cross-game generalization, and computational costs.

Opponent-Type Description Stability and Quality

W1: The paper relies on the LLM to generate opponent-type descriptions, which can be unstable and lack corresponding evaluation. The paper does not evaluate stability or quantify the generation quality. Incorrect descriptions will cause problems for downstream retrieval.

TL;DR: Our additional experiments demonstrate the consistency, retrieval stability, and accuracy of opponent-type descriptions, further confirming the effectiveness of opponent-conditioned retrieval.

Action: We will incorporate the additional results into Section 5.

We thank the reviewer for raising this important concern. We agree that opponent-type descriptions play a crucial role in OppoMem, as they guide the retrieval of relevant memories. We would like to further clarify the concept of opponent types and provide additional experimental evidence regarding their consistency and accuracy.

1. Opponent Types and Opponent Modeling

The concept of opponent types originates from game theory and has been widely adopted in type-based opponent modeling[1]. In OppoMem, an opponent type refers specifically to a natural-language hypothesis about an opponent's behavioral patterns and strategic tendencies, inferred from observed interaction histories. The effectiveness of LLM-based opponent modeling has also been demonstrated in prior work. For example, Hypothetical Minds [2] shows that LLM-based agents can construct and iteratively refine hypotheses about other agents' strategies based on observed interactions, leveraging such reasoning to improve decision-making in multi-agent environments.

2. Additional Experiments on Opponent-Type Consistency and Accuracy

To further investigate this concern, we conducted additional experiments using ten different rule-based RPS opponents with different predefined strategies.

We evaluated three aspects: (1) Description Consistency, measuring the semantic consistency of opponent-type descriptions; (2) Retrieval Consistency, measuring whether these descriptions lead to the same Top-1 memory retrieval result; and (3) Correctness, measuring whether the inferred opponent type accurately reflects the opponent's ground-truth behavioral strategy.

For the stability evaluation, we selected 20 trajectories from each of the 10 rule-based opponents and independently performed opponent-type inference 10 times per trajectory using DeepSeek-V4-Flash. An independent LLM evaluator based on GPT-6.1-sol assessed the semantic consistency of the generated descriptions, while retrieval consistency was evaluated based on their Top-1 memory retrieval results. For correctness, we evaluated the inferred opponent types against the opponents' ground-truth strategies across all 200 trajectories.

Game HorizonDescription ConsistencyRetrieval ConsistencyCorrectness
580%68.5%58.5%
1084.5%72%68%
1585%72%68%
2088.5%75.5%77%

These results provide three insights. First, opponent-type descriptions show high semantic consistency, increasing from 80% at round 5 to 88.5% at round 20. Second, Top-1 retrieval consistency remains above 68% and improves with more observations, indicating stable memory retrieval. Third, opponent-type identification achieves 58.5%–77% accuracy, demonstrating LLMs' ability to infer opponents' behavioral strategies from interaction histories. We acknowledge that imperfect opponent modeling may occasionally lead to inaccurate memory retrieval. Nevertheless, our original ablation study demonstrates the effectiveness of opponent-conditioned memory. These results suggest that, despite the imperfect accuracy of opponent-type inference, leveraging inferred opponent behavior to guide memory retrieval provides substantial benefits to downstream decision-making.

[1] Albrecht S V, Stone P. Autonomous agents modelling other agents: A comprehensive survey and open problems[J]. Artificial Intelligence, 2018, 258: 66-95.

[2] Cross L, Xiang V, Bhatia A, et al. Hypothetical minds: Scaffolding theory of mind for multi-agent tasks with large language models[C]//International Conference on Learning Representations. 2025, 2025: 6507-6546.

AuthorsOpenReview ↗

Response to Reviewer R5iv (2/3)

Cross-Game Transfer Generalizability

W2: The cross-game transfer study is small-scale and limited to one game. Adding more transfer experiments would better support the generality claims.

TL;DR: We further conducted two additional experiments, RPS→MP and IPD→MP, extending the evaluation to four transfer directions across three games. Cross-game memory outperforms Vanilla in all four settings, providing additional evidence of cross-game generalization.

Action: We will clarify the original transfer settings in Section 5.4, add the two new Matching Pennies transfer experiments, and expand the discussion of cross-game generalization.

We thank the reviewer for raising this important concern. We apologize for the lack of clarity in Section 5.4, where the setting of the transfer experiment was not explicitly specified. We would like to clarify that our original experiments already included bidirectional transfer between RPS and IPD, namely RPS→IPD and IPD→RPS. Nevertheless, we agree that evaluating transfer between only two games is insufficient to fully demonstrate the generalizability of OppoMem, and that a broader evaluation would provide stronger empirical support.

To further address this concern, we conducted two additional transfer experiments, RPS→MP and IPD→MP, extending our evaluation to four transfer directions across three games. Matching Pennies (MP) is a two-player zero-sum game in which players simultaneously choose between two actions, with opposing objectives of matching or mismatching their choices. For MP, we report the average payoff per round, normalized to a 0–100 scale. The results are summarized below.

MethodIPD→RPSRPS→IPDRPS→MPIPD→MP
Vanilla45.350.346.846.8
w/o memory48.750.147.647.6
Cross-game memory54.051.850.647.8

The results show that cross-game memory consistently outperforms Vanilla across all four transfer settings, achieving improvements of 8.7, 1.5, 3.8, and 1.0 points, respectively. Furthermore, compared with the w/o memory variant, which retains opponent modeling without stored memories, cross-game memory achieves additional improvements, particularly in IPD→RPS (+5.3 points) and RPS→MP (+3.0 points).

These findings provide further evidence that opponent-conditioned strategic memories can capture transferable behavioral knowledge beyond their originating games. As described in Section 5.4, our cross-game transfer focuses on strategic knowledge, retaining abstract opponent behavioral patterns and general response principles while adapting them to the target game's rules and legal actions. Since different games involve distinct strategic dynamics and payoff structures, the effectiveness of such knowledge transfer naturally varies across source–target game pairs. Overall, these results suggest that OppoMem can facilitate cross-game reuse of strategic experience, indicating its potential for generalization across different game settings. We will revise Section 5.4 to clarify the original transfer settings and incorporate the additional experimental results.

AuthorsOpenReview ↗

Response to Reviewer R5iv (3/3)

Token Consumption and Computational Efficiency

W3: The paper lacks a comparison of token consumption across different methods, making it difficult to determine whether OppoMem’s performance gains justify the additional cost of opponent modeling and memory refinement.

TL;DR: We conducted additional token consumption comparisons. OppoMem achieves substantial performance improvements with relatively modest additional token consumption compared with other baselines.

Action: We will incorporate the additional token consumption comparisons and performance–cost analysis.

We thank the reviewer for raising this important concern. We agree that comparing token consumption across different methods is essential for assessing whether OppoMem's performance gains justify its additional computational overhead. To address this concern, we conducted additional experiments comparing token consumption across four games using DeepSeek-V4-Flash as the backbone model.

We measured the average inference-time token consumption per action for OppoMem and the baseline methods, including both opponent modeling and action generation and additionally measured the average token consumption per experience item for methods that construct and refine experience or memories. The results are summarized below:

MethodRPSIPDHanabiDeliberationTokens / Experience
Vanilla956.81027.22674.95160.0—
ReAct1063.51239.22899.36328.7—
Reflexion1138.01327.33371.16058.94680.1
ExpeL1573.61661.53254.36852.43957.4
A-MEM1614.91650.13327.96854.64340.8
Agent-Pro1569.01730.43611.210829.4—
OppoMem1658.71790.94254.97157.54079.9

Compared with experience-based methods, OppoMem achieves better performance with token overheads of 45.8%, 5.4%, and 2.7% in RPS, and 18.1%, 4.5%, and 4.4% in Deliberation, respectively. Compared with Agent-Pro, OppoMem performs better with only 5.7% more tokens in RPS, while using 33.9% fewer tokens in Deliberation. Together with our main results in Section 5.1, these findings demonstrate OppoMem's favorable performance–token trade-off across strategic games. For experience distillation and refinement, OppoMem consumes 4,079.9 tokens per memory item, comparable to Reflexion, ExpeL, and A-MEM. Despite additional evaluation and refinement, OppoMem incurs comparable processing costs, which can also be amortized over subsequent decisions. We do not report the token cost per experience for Agent-Pro, as its policy-level reflection and optimization do not produce discrete memory items, making per-experience token costs not directly comparable.

We exclude Strategist from this comparison because it employs LLM-generated programmatic strategies combined with search-based execution, rather than relying primarily on direct LLM inference for each action. This makes its token consumption less directly comparable to the per-action inference costs of the other methods.

Overall, these results suggest that OppoMem achieves meaningful performance improvements with reasonable additional token consumption, both during inference and in memory processing. Although opponent modeling introduces computational overhead, opponent-aware memory retrieval and the corresponding response strategies provide performance benefits that can justify this cost. We will incorporate these additional results and analyses into the revised manuscript.


Typographical and Grammatical Corrections

W4: Several typos need to be addressed.

TL;DR: We acknowledge the typographical and grammatical errors highlighted by the reviewer and will correct them in the revised manuscript.

Action: We will correct all the reported typos and carefully proofread the manuscript to identify and address any remaining grammatical and typographical errors.

We thank the reviewer for carefully reading our manuscript and pointing out these typos. We apologize for these errors and will correct all the typos mentioned by the reviewer. We will also carefully check the manuscript again to identify and correct other grammatical and typographical errors.


Thank you again for your careful review and constructive suggestions. Your feedback has helped us clarify OppoMem’s opponent modeling mechanism, further evaluate the reliability of opponent-type descriptions, strengthen our experimental evidence on cross-game generalization, and provide a more comprehensive analysis of computational efficiency. We will incorporate these additional results, clarifications, and corrections into the revised manuscript.

Best,

The Authors

Dear reviewer R5iv,

Thank you again for your comments. We have posted a detailed rebuttal and a revised version of our paper addressing the concerns raised. We would greatly appreciate it if you could review these materials at your convenience and let us know if any questions remain.

Best,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Authors' Final Response

Dear ACs and Reviewers,

Thank you very much for your valuable contributions to our work. We appreciate the reviewers’ thoughtful feedback and are encouraged by the positive overall assessment, including the recognition of the paper’s novelty and contributions. In response to the insightful comments, we have conducted substantial new experiments and analyses to further strengthen the paper and improve clarity. To assist the AC and help streamline their review process, we provide below a brief summary of our contributions and the discussions with reviewers:

Core Contribution

We propose OppoMem, a novel self-evolving memory framework for LLM agents in strategic games. Distinct from all prior memory works for strategic settings, OppoMem uses opponent type as the key principle for both memory formation and retrieval. Based on this premise, it employs clustering as the core memory-bank structure, estimates memory utility via randomized activation, and adapts to changing opponents through online modeling, retrieval, and injection. Evaluated across diverse types of strategic games against multiple established baselines, OppoMem demonstrates strong performance in strategic games.

Discussions with Reviewers

For Reviewer R5iv: (Initial Rating: 3)

We have addressed Reviewer R5iv’s all four concerns by adding quantitative evaluations of opponent-description consistency, retrieval stability, and correctness; expanding cross-game transfer to four directions across three games; reporting inference and memory-processing token costs; and correcting the identified typos.

For Reviewer AUbG: (Initial Rating: 2)

Across multiple rounds, we clarified the distinctions from prior work, conducted reflection-model ablations, expanded evaluation to held-out and strategy-switching opponents, and supplied standard deviations, opponent behavioral profiles, switching schedules, and pre-/post-switch results. Reviewer AUbG explicitly confirmed that all previously raised concerns have been resolved. Reviewer AUbG nevertheless retained two objections, which we have addressed directly:

1.”OppoMem’s standard deviations overlap with at least one baseline in nearly every setting, so the results do not show that it performs better than existing methods.”

Our Response: We emphasize that overlapping standard deviations are not equivalent to a statistical significance test and, on their own, are insufficient to dismiss the observed performance gains and supporting ablation results.

2.”The additional results with strategies unseen during training show no advantage.”

Our Response: Memory-based methods are generally designed to generalize to unseen test instances by leveraging accumulated experience, rather than to solve entirely novel problems. We therefore respectfully suggest that this concern may extend beyond what memory-based methods are designed to achieve.

We maintain that these remaining points neither identify a methodological flaw nor undermine the contributions demonstrated by our experiments and analyses.

For Reviewer MHQ6: (Initial Rating: 3)

We have addressed Reviewer MHQ6’s all questions through an early-stage inference case study, representative opponent-type analysis, cross-model blind-spot evaluations, targeted-prompt experiments, and discussions of recursive modeling and multi-agent safety applications. Reviewer MHQ6 said we have addressed the reviewer’s questions.

Finally, we sincerely thank all reviewers for their constructive questions and engagement. The additional experiments and analyses conducted during this discussion significantly strengthened our paper, and we have incorporated them into the revised version.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.