acceptodds
Under review as a conference paper at ICLR 2027

COUNTERMEM: WORLD-MODEL-VERIFIED COUNTER- FACTUAL MEMORY FOR LANGUAGE AGENTS

Abstract

Existing agent memory frameworks mainly create memory through an agent’s interaction with the factual world, e.g., remembering feedback from actions taken to improve performance on future tasks. However, these frameworks seldom ask the “what if” question during memory construction: what if a different action had been taken, would the feedback have changed, and how could this feedback become useful memory? Obtaining such feedback directly in an active environment can be expensive and can alter the state needed for comparison. In this work, we introduce COUNTERMEM, a reinforcement-learning framework for constructing and using verified counterfactual memory across tasks. After a failed action, COUNTERMEM evaluates local alternatives from a copy or reset of the original state using executable world models, such as tests, proof checkers, and solvers. It stores improvements with the original and corrected actions, checked outcomes, and conditions for reuse. A learned memory-use policy selects a retrieved record or skips memory to balance task success and interaction cost, while the base LLM remains fixed. Both memory and policy are frozen during held-out evaluation. We evaluate COUNTERMEM on 12 benchmark settings across six domains. With gpt-oss-120b, COUNTERMEM improves both ReAct and Reflexion on all 12 benchmarks across six domains, averaging a gain of 12.6 percentage points over their unaugmented versions. In the four-domain comparison across two backbones, task-run tokens decrease by 7.7–42.0%, excluding offline selector-training costs. Further analyses show that removing verification or persistent storage weakens the gains, while applying verified corrections to unsuitable decisions can reverse them. Code will be released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory, Counterfactual Reasoning, LLM Agents, World Models, Test-Time Adaptation, Interactive Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ntEDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to generate synthetic "memory" with world model simulations. When the agent makes a mistake, it tries to generate alternative actions based on the experience. The world model examines whether these actions are actually helpful compared with the original one. If so, the new trajectory is logged in the agent's memory for future use. The method is demonstrated effective on multiple benchmarks and backbones.

Critical strengths
  1. The experiments are extensive and the results are strong.
  2. The presentation of the paper is easy to follow, and the method is clean and simple.
Critical weaknesses
  1. The method uses oracle world models, which are not always available in general tasks. This limits the impact of the paper.
Questions for authors
  1. Does your method work with trained world models?
Reviewer_KcJoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes COUNTERMEM, a memory framework for LLM agents that stores verified counterfactual corrections rather than only past experiences. When an action fails, the method generates a small set of alternative actions, verifies them using an external evaluator, and stores successful corrections together with applicability conditions. During future tasks, retrieved memories can be reused through a learned selection mechanism. Experiments across coding, theorem proving, formal verification, text-to-SQL, SMT, and SAT benchmarks show consistent improvements over the evaluated baselines. While the idea is intuitive and the empirical evaluation is broad, the paper leaves open questions regarding the specific contribution of each component, the dependence on executable verifiers, and the applicability of the approach beyond domains that support reliable verification.

Critical strengths

The paper introduces a clear formulation of counterfactual memory that stores verified corrective actions together with applicability conditions. The approach is evaluated across a diverse set of domains, including coding, theorem proving, formal verification, text-to-SQL, SMT, and SAT solving. The method consistently outperforms the evaluated baselines on these benchmarks, providing evidence that the proposed memory mechanism can improve agent performance in environments with executable feedback.

Critical weaknesses
  1. The contribution of the DQN selector is difficult to assess because it is not compared against simpler selection strategies.

  2. The paper does not fully isolate the contribution of counterfactual memory from other added components such as applicability conditions and the learned selector.

  3. The framework relies on executable evaluators and state reproducibility, which may limit applicability beyond the evaluated domains.

  4. The counterfactual search considers only a small number of local edits. It is unclear whether the approach extends to failures requiring larger changes in reasoning or planning.

  5. The efficiency analysis does not include offline selector-training costs, making it difficult to assess total computational overhead.

  6. The paper would benefit from more detailed analysis of memory quality, such as retrieval accuracy, applicability of retrieved memories, and the frequency of harmful memory reuse.

Questions for authors

Check my comments in Weaknesses.

Reviewer_VrzcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents CounterMem, a framework that builds agent memory from verified counterfactual corrections and learns when to use them. During memory construction, when an action fails, the agent proposes a few local edits, such as changing one SQL condition or proof step, and evaluates each from the same state with the task's executable checker, which the paper calls a world model. Edits that improve the outcome are stored with the original action, the check results, and the conditions under which the correction applies in the memory store. A deep Q-network (DQN) selector is then trained on separate tasks to pick one retrieved record or skip memory. At inference, the LLM drafts an action and revises it with the retreived memory record, if any, while the LLM, store, and selector stay frozen. The authors show that CounterMem improves both ReAct and Reflexion on mathematics, coding, formal verification, text-to-SQL, SMT, and SAT benchmarks while using fewer task-run tokens.

Critical strengths
  1. The paper's use of counterfactual memories is a sensible extension of experience memories such as ExpeL
  2. The paper's RL formulation for the retrieval problem is pretty interesting and there can be value in developing this modeling for future work.
  3. CounterMem improves both ReAct and Reflexion with both backbones while using fewer task-run tokens.
Critical weaknesses
  1. DQN selector abations are missing and the steps that decide what reaches it are under-specified.: The paper does not compares the DQN selector with even simple basleines with top-1 retrieval, a DQN without the skip option, or an LLM choosing the record.
  2. The paper skims over critical implementation details:
  • (a) How is duplicate and conflict detection done during memory creation (R and I in Eq. 4)?
  • (b) How is it determined whether a condition is "compatible" with a new task (Eq. 7)?
  1. The baseline setup is unclear. CounterMem gets a separate verification budget while building memory but the paper does not report the baselines' budgets, and the adapted baselines often score below the no-memory agent (38 of 64 cells in Table 1). The accumulation, transfer, and equal-budget studies then switch to ReasoningBank, which is not in Table 1, without saying why, how it was set up, or which agent and backbone they use.
  2. The positioning omits close prior mechanisms. AutoGuide [1] stores guidelines with the context in which they apply, ICAL [2] turns suboptimal trajectories into corrected examples, and REMEMBERER [3] updates an experience memory with reinforcement learning while keeping the LLM frozen. Since the paper doesn't contrast against these papers, it is unclear what CounterMem's records and selector add beyond them.

References [1] Y. Fu et al. AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents. NeurIPS 2024. https://arxiv.org/abs/2403.08978 [2] G. Sarch et al. VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought. NeurIPS 2024. https://arxiv.org/abs/2406.14596 [3] D. Zhang et al. Large Language Models Are Semi-Parametric Reinforcement Learning Agents. NeurIPS 2023. https://arxiv.org/abs/2306.07929

Questions for authors

Q1. On the frozen stores, how does the DQN compare with always using the top-ranked retrieved record, with the DQN without the skip option, and with letting the LLM choose a record or none? At test time, how often does the selector skip, pick the top-ranked record, or pick another one?

Q2. How are new source tasks added after deployment, and does the selector then need retraining? Fig. 2 reuses one trained selector across all memory snapshots which store was it trained on?

Q3. Why is ReasoningBank used in the accumulation, transfer, and equal-budget studies but not in Table 1? How was it built: which source tasks, which feedback signal, and which backbone? Which agent–LLM pair do Figs. 2, 3, 7, and 8 use?

Q4. For each Table 1 row, how many source tasks, evaluator calls, and tokens went into building memory? Did the baselines see evaluator outcomes? What do memory construction and selector training cost for CounterMem? How many runs underlie Tables 1 and 6, and what are the standard deviations across the five selector seeds? The 12-benchmark result also covers one backbone and reports only gains, several of them small (+2.5 to +3.8 for Reflexion); could you add absolute scores?

Q5. How do CounterMem's records and selector differ from AutoGuide's conditional guidelines, ICAL's corrected examples, and REMEMBERER's RL-updated experience memory?

Minor comments (not affecting my assessment)

  • In Fig. 8(b), the "ordinary tasks" harmful-use rates for ReasoningBank and CounterMem (9.4% and 4.2%) match the "same error, new surface" values in Fig. 3(c). Please state which tasks Fig. 8(b) uses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.