acceptodds
Withdrawn from ICLR 2027

CARE-Mem: Decision-Level Counterfactual Credit for Schema-Free Memory Revision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agents with long-term memory must update outdated information while preserving useful memories. However, a single reward for an entire memory update can reinforce unnecessary edits alongside correct revisions. This paper introduces CARE-Mem, which learns how to revise memory using decision-level counterfactual credit. Given supplied or retrieved memories, CARE-Mem predicts node actions and propagation gates without predefined domain slots or an ontology. During training, it executes counterfactual proposals from the same memory snapshot and assigns the resulting utility differences to the corresponding decision spans in the sampled proposal. At inference, CARE-Mem executes a single proposal without counterfactual evaluation. This study also introduces CAREBench-240, a human-annotated benchmark for node actions, propagation gates, and replacement text. CARE-Mem achieves higher action and edge prediction scores than graph-memory baselines mapped to the same interface. In post hoc exploratory comparisons, CARE-Mem achieves strict replacement exact-match accuracy of 48.68%, versus 39.57% for the parent policy trained by supervised fine-tuning (SFT) and 40.77% for the global-plus-local control. On MemoryAgentBench FactConsolidation, CARE-Mem averages 44.00% accuracy across single- and multi-hop tasks, exceeding the SFT parent and global-plus-local control by 11.00 and 9.17 percentage points, respectively, without further parameter updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-term memory, memory revision, counterfactual credit assignment, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sXKKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CARE-Mem, a memory revision policy trained using decision-level counterfactual credit assignment. The policy outputs node actions, replacement text, and propagation gates. During training, single-node, single-edge, and local joint interventions are executed from the same memory snapshot, and the resulting utility differences are assigned to the corresponding output tokens. At inference, only one proposal is executed, without counterfactual evaluation. The authors introduce CAREBench-240, a human-annotated dataset containing 240 cases, and evaluate transfer performance on MemoryAgentBench FactConsolidation. CARE-Mem outperforms graph-memory baselines mapped to the same interface. Its main improvements over the SFT parent and global-plus-local control are in replacement-text accuracy and downstream question answering.

Critical strengths
  1. The paper proposes a concrete, executable credit-assignment scheme to address the limitations of a shared reward for an entire proposal. Modifying node actions or propagation gates from the same snapshot allows the method to assess a decision’s effect on execution and assign the signal to the corresponding output span. Counterfactual controls use deterministic execution and require no additional model generations, while inference incurs no counterfactual overhead. This gives the method clear technical substance and practical value, which is the main reason for supporting acceptance.

  2. Matched controls and external evaluation provide evidence of effectiveness. With the same SFT initialization, training cases, and decoder, strict replacement accuracy increases from 39.6% to 48.7%. With the learned adapters frozen, average MemoryAgentBench accuracy increases from 33.0% to 44.0%. CAREBench also uses independent annotation and adjudication and reports high inter-annotator agreement. These results suggest that the gains are not solely attributable to interface adaptation or additional supervision.

Critical weaknesses
  1. The existing comparisons do not fully establish that the gains come from more accurate decision-level credit assignment. In Table 3, CARE-Mem’s Action Macro-F1 and Edge F1 are approximately the same as those of the SFT parent, and neither metric differs significantly from the SFT parent or A2. The main improvement is in replacement text. REVISE accounts for only 15.4% of node decisions but receives 50.4% of the absolute node token-credit mass because replacement text makes its spans longer. Moreover, global advantages are standardized, local credits are not, and A2 directly adds the two. The comparisons therefore change the credit source, numerical scale, and relative training weights of different output components simultaneously. The gains may partly reflect differences in optimization scale or greater weighting of replacement text. This limits attribution of the gains to the paper’s core algorithm.

  2. The relationship between the utility function and revision correctness is problematic. Executing the human-adjudicated reference proposals yields average Current, Retention, and History scores of only 0.5625, 0.4792, and 0.5000 in Table 22. The authors acknowledge a mismatch between these scores and structural correctness. In Table 21, 75.25% of edge controls change only edit cost, while History remains unchanged across all interventions. Removing the Retention term (A5) produces the highest mean Action Macro-F1 and replacement accuracy, although the Macro-F1 difference is not statistically significant. These findings raise the possibility that local credit primarily reflects edit-cost preferences and scoring rules rather than revision utility. Since counterfactual credit directly depends on this utility function, the issue concerns a central assumption of the method.

Questions for authors
  1. Why do correct reference proposals receive probe scores substantially below one? How much of this comes from empty probe families, and how much from scoring or execution mismatches? Do these issues also occur in the 16 cases used for policy training?

  2. What results are obtained when local credit is generated from edit cost alone? Given that most edge controls change only cost, could this baseline explain the observed edge credit and downstream gains?

  3. After matching advantage scales and ensuring that each decision’s total training weight is independent of its token count, does CARE-Mem still outperform global-only and global-plus-local training? This would directly affect attribution of the gains to credit assignment.

  4. Node counterfactual controls bypass reachability, whereas the original proposal follows gating. For an unreachable node, the utility difference therefore changes both the action and whether it is executed. Why should this difference be assigned to the node action rather than the propagation gate?

  5. The 800 MemoryAgentBench questions share four fact contexts. Is the average gain of 11 percentage points consistent across these contexts? A2’s Run 0 is comparable to CARE-Mem, while its other two runs perform much worse. Does this mainly reflect training instability or revision errors in particular contexts?

Reviewer_Wh3eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1(−1)
Presentation
2
Soundness
1
Read review
Paper summary

The authors look into how agents with long-term memory should revise their memories when new information arrives. They argue that a single reward for the whole update can credit unnecessary edits along with correct ones.

The authors propose CARE-Mem, which represents an update as a structured proposal: an action for each memory item (keep, invalidate, revise or uncertain), replacement text, and gates on dependency edges that control propagation. During training, it executes counterfactual proposals from the same memory snapshot and assigns the utility differences as credit to the corresponding decisions. At inference, it executes a single proposal. They also introduce CAREBench-240, a human-annotated benchmark of 240 cases.

Some findings: starting from a Qwen3-4B SFT parent, CARE-Mem outperforms graph-memory baselines mapped to the same interface, and in post hoc comparisons gets higher strict replacement accuracy than the SFT parent (48.7% vs 39.6%). On MemoryAgentBench FactConsolidation, it reaches 44.0%, 11 points above the SFT parent.

Critical strengths
  1. The formulation is clean. It separates node actions from propagation gates makes it explicit which decision is right or wrong.
  2. CAREBench-240 is annotated and could be a useful resource.
  3. The protocol is transparent, with primary and exploratory metrics declared upfront and an honest limitations section.
Critical weaknesses

There are a few things that I want to point out:

  1. The RL stage does not seem to improve over the SFT parent on the paper's own primary metrics. In Table 3, CARE-Mem gets 65.5 / 60.7 Action Macro-F1 / Edge F1 versus 65.7 / 60.7 for the SFT parent. The simpler global-plus-local control is slightly higher (67.2). So the large gap over A-MEM and CUPMem in Table 1 mostly comes from SFT on interface-matched data, but Table 1 does not include the parent. The same holds on MemoryAgentBench, where the parent already gets 33.0 versus 19–22 for long-context and retrieval baselines. The remaining gains are on metrics the paper itself labels exploratory or descriptive, so the counterfactual credit in the title is not what produces the main numbers.
  2. The training reward does not seem to recognize correct revisions. In section 4, it notes that the history score never changes during training, and section 6 reports that most edge-credit differences come from edit cost alone. So at least for edges, the credit seems to mostly reflect edit cost rather than correctness, which may explain why removing edge counterfactuals does not hurt edge prediction and removing the retention probe does not hurt at all. I think the probes need to be fixed before the credit can be trusted.
  3. The RL stage is way too small to support claims about credit assignment. Training uses 16 cases, two rollouts per case and two optimizer updates in total. With two rollouts, many groups are tied and give no signal, and the per-run results vary a lot across seeds, so it is hard to tell whether a condition is better because of its credit scheme or because of its seed.
  4. The setting is hard to scale. Training requires case-specific probe questions with known answers, and all systems are given the candidate memories and their relation labels, so retrieval and relation typing are never evaluated. It is not clear how this would work for a real agent with a large, evolving memory in real deployment environment.
Questions for authors
  1. With more updates and at least five seeds, does CARE-Mem beat the SFT parent and the global-plus-local control on the primary metrics?
  2. If the probes are fixed so that reference proposals score close to 1, does decision-level credit still help?
  3. How would the method work without supplied candidates, relation labels or probe answers? Could you also add the SFT parent to Table 1?
Reviewer_7TvaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARE-Mem trains an LLM policy to revise an agent's memory without predefined domain slots. It receives an observation plus supplied candidate memories linked by relation-labelled edges. It then emits one proposal: a KEEP / INVALIDATE / REVISE / UNCERTAIN action for each memory, replacement text for each REVISE, and a propagation gate for each edge. Only actions reachable through open gates are executed, and old versions are kept. During training only, the update is re-run from the same snapshot with one action, gate or small connected set changed. The resulting utility differences (probe scores minus an edit cost) replace the proposal-level advantage on those decisions' tokens in a PPO-style update. A Qwen3-4B SFT parent is trained this way on 16 cases for two updates. The paper also introduces CAREBench-240, which has 240 synthetic, human-annotated cases.

Critical strengths

S1. Exact, low-cost credit for each decision. Controls re-run the update from the same pre-write snapshot after changing one action, one gate, or a small connected set of decisions. They are deterministic executions, not extra model generations. Training costs about the same as global-only training (0.556 vs. 0.552 GPU-h), and inference makes no counterfactual calls (Table 5).

Related memory-RL methods assign credit more coarsely: Memory-R1 per outcome, HiMPO per memory write, TreeMem per agent (sampled), MemBuilder per memory component. CARE-Mem scores each action and gate within a single update. Its node credit separates aligned from misaligned decisions (AUROC 0.864). It also shows the problem it targets: 11 of 43 rollouts with positive proposal-level advantage contain a wrong decision, and local credit is negative for 68% (110/161) of wrong decisions (Sec. 6.4).

S2. Gains over the SFT parent, including on an external benchmark. CARE-Mem beats the parent on two measures, while keeping its Action Macro-F1 and Edge F1:

  • strict replacement accuracy: +6.5 to +12.2 points (Table 15);
  • MemoryAgentBench FactConsolidation, with frozen adapters: +10.5 to +11.9 points (Table 10).
Critical weaknesses

W1. The paper doesn't show that decision-level credit is what helps. On the endpoints the authors pre-specified, CARE-Mem is indistinguishable from its own SFT parent: Action Macro-F1 moves by −0.15 points and Edge F1 by +0.02 (Holm p = 1.0, Table 8). The large margins over A-MEM and CUPMem in Table 1 therefore come from fine-tuning, which is hard to see because Table 1 leaves out the parent row. The significant gains over the parent and A2 appear only in post hoc strict replacement accuracy. Even there, the variant without the Retention probe (A5) does as well or better, and it also has the best Action Macro-F1 (67.8).

The comparison with global-only training (A1) is harder to interpret than it looks. In CARE-Mem's own training rollouts only 43 of 192 have positive advantage, which with K = 2 means 53 of 96 groups tie. CARE-Mem's local credit, by contrast, is unnormalized, and about half of the node credit lands on REVISE spans. The two runs therefore differ in several ways at once, as the authors acknowledge in Sec7. On MemoryAgentBench, which is reported only descriptively, A1 is missing altogether, and A2 matches CARE-Mem in Run 0 (43.88% vs. 43.63%).

W2. The utility that drives the credit is only loosely tied to correctness. The adjudicated reference proposals score only about 0.5 on each probe family, and 0.0 on Current for INVALIDATE cases (Table 22). One case that is correct in every respect scores 0/0/0 (Table 12). History never changes across 2,376 interventions. In three quarters of the edge controls, the only thing that changes is the edit cost, so for misaligned edges the credit is negative barely more often than chance (53.9%).

The results are consistent with this. CARE-Mem's INVALIDATE recall drops below the parent's (0.844 vs. 0.896). Training without edge credit actually gives a higher Edge F1 (61.9 vs. 60.7).

Questions for authors

Q1. Does the advantage over proposal-level credit survive a fairer comparison? Please train at least 5 seeds per condition and use more than two updates. Compare CARE-Mem against A1 with K ≥ 4 (or with advantages rescaled to match local credit), and against a control that shuffles local credit across decisions. With K = 2, A1's advantage is just a sign, and the current p-values hold the trained models fixed. If CARE-Mem still wins under these conditions, the central claim would be much more convincing.

Q2. On MemoryAgentBench, how are the "claim keys" computed? What do A1, and a simple rule that invalidates every same-key predecessor, score under the same pipeline? FactConsolidation tells the agent that newer facts win, so grouping by key may already do most of the work. A slot-like key would also sit uneasily with the "schema-free" claim.

Q3. Why do the adjudicated reference proposals score only about 0.5 per probe family, and 0.0 on Current for INVALIDATE cases? Were the training cases scored the same way? How often does the utility rank the correct decision below its counterfactual control?

Q4. How well would a simple rule using the supplied relation labels do on CAREBench (e.g., open a gate whenever the label is not UNRELATED)? And what exactly is the "Direct Revision" baseline, which yields under 1% valid outputs? Without a reference point like this, it is hard to tell how much of the structural score reflects learning rather than label reading.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.