acceptodds
Withdrawn from ICLR 2027

From Recurrence to Consequence: Risk-Aware One-Shot Memory Consolidation for Long-Term LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-term memory is essential for LLM agents operating across multi-session interactions, but existing memory systems primarily optimize for information that is frequent, salient, or generally useful. This leaves a critical failure mode: an interaction may appear only once, yet forgetting it can lead to unsafe, non-compliant, or irreversible downstream actions. We study this setting as rare-but-critical one-shot memory, where the value of a memory is determined not by recurrence, but by the consequence of forgetting. We first show that recurrence-driven systems exhibit a substantial one-shot consolidation gap, while broad-admission methods improve recall at the cost of false promotion, over-triggering, and large operational overhead. To address this problem, we propose ROMC, a risk-aware memory consolidation framework that promotes one-shot information into critical memory when its omission may cause high-impact future failures. ROMC estimates consequence-aware risk using structured risk dimensions and counterfactual future probes, and distills this teacher judgment into lightweight student scorers for efficient write-side promotion and read-side activation. Unlike standard memory QA, we further introduce an action-oriented downstream evaluation that requires agents to generate safe operational advice or alternatives under remembered constraints. Experiments show that ROMC achieves the best overall downstream correctness among deployable methods, while substantially reducing over-triggering, injected constraints, and token cost. These results suggest that long-term agent memory should move beyond remembering what recurs, toward consolidating what would be costly to forget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Long-Term Memory, Memory Consolidation, Risk-Aware Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1vkGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets "rare-but-critical one-shot memories", constraints that are stated once but whose omission can cause unsafe or non-compliant actions, and argues that recurrence- and salience-based admission systematically misses them. ROMC uses a teacher LLM to annotate each utterance with a five-dimensional ordinal risk vector and with trigger/boundary/non-trigger probes carrying remember-vs-forget deviation and severity scores. These are aggregated (Eqs. 3–5) into a PROMOTE/PROBATION/DISCARD decision and distilled into linear heads over a frozen BGE-M3 embedding, with teacher fallback near decision boundaries. A separate logistic-regression activation head controls read-time hard/soft injection (Table 1). Evaluation covers a 90-item write-side benchmark (NatRisk, Table 2), a 1,000-task end-to-end benchmark (Final-1000, Table 3), and extensive appendix analyses.

Critical strengths

Problem framing and evaluation design. Separating raw retention from durable hardening (App. B.3), and pairing trigger tasks with single-entity confusion tasks (Sec. 4.4, App. C.1), exposes a trade-off that memory QA hides. Broad extractors reach high recall only with FPR 0.683–0.950 (Table 2) and OTR of 0.126–0.182 (Table 3). This protocol is reusable independently of the method. Deployability. The student takes about 18 ms per utterance versus 12.5 s for the teacher (App. E.1) and escalates about 16% of items (App. C.6). Downstream, it injects 3.99 memories per task versus 5–48 for the baselines, and it has the lowest operational tokens of any deployable method.

Critical weaknesses

The "counterfactual cost of forgetting" is asserted, not measured, and its contribution is not isolated. Eq. 2 is described as rollouts of agent F with and without m_t. In practice, the teacher sees only the isolated sentence (Fig. 1). In the same call that produces the structural ratings, it writes hypothetical with/without-memory behaviors and assigns them rubric scores (App. F, Steps 5–6), and M_t never enters. At deployment, S_probe is a linear regression from the same utterance embedding as S_struct (Sec. 3.3). The ablation (Sec. 4.3) is scored against teacher labels computed by the full Eq. 4–5 formula, so the "complementarity" result is partly circular. Even so, Fig. 25 shows Structure-only exceeding Full in F1 (0.783 vs. 0.762), and "no transient gate" identical to Full (0.762 / 0.089). This contradicts the text (FPR 0.095 / 0.122, and "Full achieves the best F1"). The evidence supports a distilled LLM risk rubric, not consequence-aware consolidation.

Downstream claims are overstated and not attributable to write-side admission. On Final-1000, ROMC's Strict CR (0.718) is below A-MAC (0.740) and RecMem-full (0.792). Its Overall lead (0.824 vs. 0.808, i.e., 16 of 1,000 tasks, single run, no CIs) comes entirely from lower OTR, because Overall weights trigger and confusion tasks equally. That weighting runs against the paper's asymmetric-cost motivation: with a 2:1 weight on trigger tasks, RecMem-full ranks first. The OTR advantage plausibly comes from the read-side head, which is trained on teacher probes built the same way as the confusion tasks, while the baselines inject a fixed top-k (Mean Inject exactly 5, 10, 25 or 48). No component-swap study is reported, and App. C.4 itself attributes the ranking to the read policy. On the write side, ROMC still misses 13 of 30 critical items (Recall 0.567, Table 2), which is exactly the failure mode the paper targets. Finally, RecMem-full achieving the best deployable Strict CR suggests that the claimed recurrence "consolidation gap" is largely an artifact of durable-only accounting.

Questions for authors

1: Are any with/without-memory behaviors in Eq. 2 obtained by executing a downstream agent? If not, please compare probe scores from real rollouts against the structural rubric alone, at matched teacher cost, using ground-truth rather than teacher-derived labels.

2:Table 2 reports 548 online tokens for 600 turns, whereas ~16% escalation with a teacher prompt of over 2k tokens implies far more. Was fallback active in Tables 2–3, and how are PROBATION records counted under durable-only scoring?

Reviewer_qs9cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ROMC (Risk-Aware One-Shot Memory Consolidation), a framework for preserving rare but critical information in long-term LLM agent memory. ROMC estimates the consequences of forgetting information through structured risk assessment and counterfactual future probes, then distills these judgments into lightweight models for memory admission and query-dependent activation. Experiments demonstrate improvements in downstream correctness and reduced memory over-triggering and computational costs.

Critical strengths
  • The paper identifies an important limitation of existing memory systems, highlighting that infrequent information can have significant downstream consequences if forgotten.
  • The proposed framework combines consequence-aware memory admission, teacher-student distillation, and selective memory activation in a coherent design.
  • The experiments evaluate both memory admission and downstream agent behavior, providing useful evidence of improved selectivity and efficiency.
Critical weaknesses
  • The methodological novelty is somewhat limited, as the framework primarily combines structured risk scoring, counterfactual evaluation, and teacher-student distillation.
  • The admission recall of ROMC is relatively low (0.567 in Table 2), raising concerns about missing critical information in safety-sensitive applications.
  • The evaluation relies on relatively small, controlled scenarios and does not sufficiently demonstrate generalization to realistic long-term agent interactions.
  • The efficiency comparison excludes offline teacher annotation and student training costs, making the overall computational advantages less clear.
Questions for authors
  • How does ROMC compare with simpler risk-aware admission classifiers under matched training data and computational budgets? Given the relatively low recall of critical memories, how can ROMC reduce the risk of missing high-impact constraints?
  • How robust is the counterfactual risk estimation to inaccurate or unrealistic future scenarios generated by the teacher model?
  • How does ROMC handle conflicting, outdated, or updated constraints in continuously evolving agent memory?
Reviewer_y3DBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies rare-but-critical one-shot memory in long-term LLM agents, meaning constraints stated once (an allergy, a dual-authorization rule) whose omission later causes an unsafe or irreversible action, and argues that existing admission policies based on salience (Mem0), recurrence (RecMem, MemoryOS), or generic utility scores (A-MAC) do not estimate the consequence of forgetting. The proposed method, ROMC, defines a memory’s value as the expected downstream loss from forgetting it (Eq. 1) and approximates it with a teacher LLM (Claude Sonnet 4.5) that rates five ordinal risk dimensions, writes matched trigger, non-trigger, and boundary future scenarios, and scores behavioral deviation and consequence severity for each, after which a weighted aggregate is thresholded into PROMOTE, PROBATION, or DISCARD. The teacher’s outputs are distilled into a lightweight student (frozen BGE-M3 encoder with CORN ordinal, probe-regret, and relevance heads) with teacher fallback near decision boundaries, and a separate read-side logistic-regression head decides whether a stored memory is injected as a hard constraint, an advisory constraint, or dropped. On the author-built NatRisk benchmark (30 positives, 60 negatives), ROMC reports the highest durable-admission precision (0.515) and balanced accuracy (0.650) with FPR 0.267 versus 0.683 for A-MAC, F1 0.540 versus 0.566 for A-MAC, and 548 online tokens. On Final-1000 (500 trigger and 500 confusion tasks), it reaches the highest Overall correctness among deployable systems (0.824 versus 0.808 for RecMem full) with an over-trigger rate of 0.072 and roughly 500K operational tokens. The appendix adds read-side analyses (raw cosine AUROC 0.62 versus distilled head 0.87), threshold sensitivity, latency (18 ms versus 12.5 s per utterance), and a fixed-bank Downstream-244 comparison.

Critical strengths
  1. The core idea is pretty interesting. Instead of asking whether a memory is salient or repeated, the method asks what breaks if the agent forgets it, and then actually builds the pipeline around that question with matched probes, tail-sensitive aggregation, and a probation tier for things it is not sure about yet.
  2. The distillation is practical. Distilling the teacher’s structured outputs rather than a yes/no label means thresholds can be retuned without retraining, and each decision can be inspected.
Critical weaknesses
  1. NatRisk only has 30 positives and 60 negatives, I am not sure how dependable the results are from testing on this benchmark.
  2. RecMem (full) and A-MAC both enforce more critical constraints than ROMC on Final-1000 (Strict CR 0.792 and 0.740 versus 0.718), , so the “one-shot consolidation gap” only shows up under the durable-only scoring convention. ROMC’s overall win seems to come from triggering less often rather than remembering more.
Questions for authors
  1. The write-side student sees only x_t with no dialogue history. What fraction of NatRisk and Final-1000 constraints require prior context to interpret, and how does ROMC perform on those?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.