acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Failure Regimes in LLM Agent Memory with Extraction-Pass Disagreement

Abstract

LLM agents build long-term memory by consolidating interaction history into a structured store, and this consolidation is lossy in two ways. In the first, a stochastic pass drops information that another pass may keep, and computation removes the error. In the second, a capability-limited backbone compresses history into a low-rank sketch, and the bias it introduces is beyond the reach of computation. The two failures look the same in aggregate accuracy, so existing systems apply the same remedy to both. We propose Regime-Routed Construction (RRC), a training-free configuration layer whose routing follows the diagnosed failure regime. RRC repeats consolidation on the same history and reads how much the passes disagree. The disagreement estimator, combined with backbone capability, routes the vote budget, the retrieval depth, and the answer policy. A noisy reading brings more extraction passes merged by majority vote. A quiet reading deepens retrieval and calibrates the answer policy when the backbone sits above its capability threshold, and declines what the store cannot support below it. A Fano-type argument shows that below a capability threshold, error on relational questions is irreducible. Across two benchmarks and backbones from 1.5B to frontier API models, RRC improves on the strongest published LoCoMo result by +4.7 F1 and outperforms the RL-trained Memory-R1 at matched backbones without gradient updates. On LongMemEval, the gains are small and insignificant, consistent with a diagnosis of little headroom.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-term memory, memory consolidation, extraction disagreement, test-time compute, retrieval, information theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AcvBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that failures in LLM-agent memory construction fall into two regimes: (i) random sampling noise, which can be eliminated by repeatedly sampling from the same history and taking a majority vote; and (ii) representational loss caused by a capability-limited backbone, which cannot be eliminated and can only be handled by abstention. Based on this distinction, the authors propose RRC, a training-free routing layer that runs kk extractions over the same window, computes inter-run disagreement (the average pairwise Jaccard distance between extracted node sets) as an estimator q^\hat{q}, and, together with a capability indicator (log⁡\log parameter count) and a question reasoning-depth coordinate dd, routes among four configurations (add voting, deeper retrieval, decline policy, and default). The paper provides a sketch formal model and a Fano-type argument establishing the existence of a capability threshold and the property that the estimator tracks only noise, not bias. Experiments are conducted on LoCoMo and LongMemEval.

Critical strengths
  • S1. The core idea is clean and well motivated. Treating the stochasticity of the construction pipeline itself as a free diagnostic signal, and moving the consistency signal from the answer level (self-consistency, semantic entropy) to the construction level in order to diagnose Regime I, is a reasonable and non-obvious move.
  • S2. The formalization and proofs are self-consistent. Propositions 1–6 are standard uses of tools such as Hoeffding/Fano/U-statistics and capture the correct invariants; the cross-capability gain/loss patterns are qualitatively consistent with the two-regime story on the numbers reported by the authors.
Critical weaknesses
  • W1. Within-group rankings in the main experiments do not control for backbone, and the “bold is best, underline second best per group” convention actually presents cross-backbone rankings. In the Small group (1.5B–3B), GAM is run on Llama-3.2-3B (36.27), Synapse is run on Qwen-1.5B (35.9) and Qwen-3B (36.6), and the A-Mem/MemoryBank/MemoryOS rows are cross-backbone reruns of Synapse on Qwen-1.5B/Qwen-3B (B.1), whereas the authors’ own Baseline/RRC rows are run on Qwen2.5-3B and Llama-3.2-3B. The Medium group mixes Qwen2.5-7B, LLaMA-3.1-8B, and Qwen2.5-14B; the Large group mixes a single Long-context baseline row (Llama-2-70B-Chat, 17.9) into Qwen2.5-72B rows; and in the API group all competitors are GPT-4o/GPT-4o-mini while this method uses DeepSeek-V3. This directly conflicts with the statement in §4.1 that “Table 1 compares against published memory systems at matched backbones.” The headline in the abstract and conclusion that exceeds Kumiho (57.8/58.5 vs. 56.5) is a cross-backbone comparison of DeepSeek-V3 against GPT-4o and cannot support conclusions about method superiority.
  • W2. The same column of the main table mixes incomparable scoring protocols. The authors’ recipe rows average four non-adversarial categories, while the headline rows average the official five categories (Table 1 caption); GAM and Synapse published values are four-category averages excluding adversarial (B.1); the remaining published values are self-reported overall token F1 from their respective papers; Kumiho’s 56.5 includes an adversarial score of 97.5 (its pinned reproducible value is only 53.1, B.1); and Memory-R1’s LongMemEval numbers (46.70/45.20, †) use its self-selected subset and scoring protocol but are placed in the same column as the authors’ weighted micro-F1 and included in within-group rankings. On the one hand, the authors state that “Judge scores are not comparable across judges, and we do not rank against leaderboard entries scored by other judges” (B.1), and that LongMemEval judge scores are “not comparable across systems” (§4.2); on the other hand, Table 1 presents exactly such a mixed ranking.
  • W3. The core paired comparison is asymmetric in answer policy, and other configuration dimensions are also asymmetric. §4.1 (l.366–367) states that the baseline uses “the gentle answer policy” and that “RRC differs only in vote, estimation, and routing,” but the Table 10 footnote reveals that “the RRC recipe rows run the plain answer policy,” and Table 12 shows that, on the same RRC framework, gentle is clearly lower than plain (3B: 40.26 vs. 43.39; 7B: 37.22 vs. 43.22). Therefore, the paired gains reported in the main text (e.g., 3B +3.79, 7B +5.29) likely come mainly from the answer-policy mismatch rather than from the RRC mechanism itself. The largest gain, 8B +9.93, further stacks a bge-base retriever upgrade applied only to RRC rows (Table 10 footnote, C.1). Moreover, each backbone reports its own best arm: 3B/7B choose plain, 8B chooses the bge upgrade (46.68), 14B chooses the grounded policy (45.05), Llama-3.2-3B chooses grounded (on this backbone the plain gain is only +0.08, C.1), and the API headline chooses gentle+top-75 (l.380–381); by contrast, each compared published system has only one fixed configuration. Grounded is not among the four routing paths in Table 4, and the bge-base upgrade is not a routing parameter; both conflict with “differs only in vote, estimation, and routing.”
  • W4. The paper’s own ablations undermine its core mechanism. The core prescription for Regime I—increasing the number of extraction votes—has no measurable effect: k=1/3/5k=1/3/5 yields 40.59/40.26/40.09, a range of 0.50 within the σ=0.45\sigma=0.45 noise; the paper itself admits that “the merged store is vote-neutral; votes do not denoise it directly” (§4.4). Table 2 further shows that replacing majority voting with union merging yields a higher 3B result (43.75 vs. 43.39) and a roughly equal 7B result (43.07 vs. 43.22). The only ablation that produces a large difference is the decline clause (replacing it with verify-then-decline collapses to 30.12/30.33), but that is an answer policy, not the estimator or routing. The paper also does not provide an ablation of the router itself (e.g., forcing routing to each path). Therefore, the mechanism claimed as the core contribution—the estimator plus routing—is not shown by the ablations to contribute any part of the gain; combined with W3, the observed gains are more likely explained by policy switching and the retriever upgrade.
Questions for authors
  • Q1. Can the authors rerun the main baselines under a unified harness and a unified backbone, and provide a truly comparable main table?
  • Q2. Please add results for {baseline, RRC} × {plain, gentle} on at least two fixed backbones, and report the actual gains under a matched answer policy.
  • Q3. Please eliminate the asymmetries across configuration dimensions in the core comparison: do not perform post-hoc arm selection per backbone, do not upgrade the retriever on only one side, and report corrected gains. In addition, please add a routing ablation (forcing each path vs. a fixed configuration) to separate the contributions of the estimator and the routing mechanism itself.
1 comment

We thank the reviewer for the careful review.

W2. As noted in our reply to Reviewer 2, the revision regroups the main table by answer contract and separates the retrieval protocols.

W3. We agree that the retrieval-side comparison should be run under a single harness. The revision reruns all single-pass baselines with the same reader, encoder, list depth, and answer contract, and reports paired differences with confidence intervals.

W4. The ablations are intended to isolate the configuration layer. The revision reorders them to make this role explicit and adds the routing ablation under the unified harness.

We thank the reviewer for the comments, which have directly shaped the revision plan.

Reviewer_UGiiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that memory construction in LLM agents suffers from two kinds of loss: extraction noise that repeated passes can remove (Regime I), and capability-limited representational loss that no amount of computation can recover (Regime II). It proposes RRC, a training-free layer that runs multiple extraction passes with majority voting, uses the mean pairwise Jaccard distance between passes as a noise estimate q̂, and combines it with a capability index (log parameter count) and a theoretically derived capability threshold to route the system to one of four configurations: more votes, deeper retrieval, declining, or default. The theory provides an exponential bound for majority voting and a Fano-type lower bound on irreducible error. On LoCoMo, RRC improves over the baseline by 1.2–9.9 F1 for backbones of 3B and above, with a headline of 57.8/58.5; gains on LongMemEval are not significant.

Critical strengths
  1. The distinction between noise recoverable by computation and irreducible loss due to limited capability is conceptually clear and useful for memory-system design.
  2. The method is training-free, is evaluated across many model scales, and reports paired bootstrap confidence intervals.
  3. Negative results are reported candidly: performance drops at 1.5B, and LongMemEval gains are not significant.
Critical weaknesses
  1. The main gains may come from the answer policy rather than diagnosis and routing. Section 4.1 states that the baseline and RRC differ only in voting, estimation, and routing, yet the baseline uses the declining gentle policy while the main RRC rows use the non-declining plain policy (Table 10). Moreover, the main-table average excludes the adversarial category, which is exactly where declining pays off. Table 12 shows that switching gentle to plain alone gives about +3.1 and +6.0 F1 at 3B and 7B, roughly matching RRC's gains over the baseline (+3.79, +5.37).

  2. The core mechanism lacks empirical support. Removing majority voting has almost no effect (3B: 43.75 vs. 43.39), and vote budgets k=1/3/5 give the same results (40.59/40.26/40.09), so the noise path does not appear to contribute.

  3. The validity of q̂ is questionable. All readings fall between 0.54 and 0.82, and the largest (API) model reads the noisiest (0.817). The quiet/noisy cut is never stated (on LongMemEval, 3B at 0.692 is called quiet while 1.5B at 0.754 is noisy). A Jaccard distance over node names may mostly reflect naming variation rather than information loss; Figure 2's own disagreement example includes a node renaming.

  4. The theory is disconnected from the method. κ, s_d, and g(C) are never measured, and no value or procedure for C* is given, so the threshold appears to be placed post hoc. Also, 1.5B is diagnosed as below threshold and should take the decline path, yet its RRC adversarial score is 0, suggesting the plain policy was actually run.

  5. Inconsistent comparison protocols. The main table mixes four-category and five-category averages; the headline uses top-75 retrieval and the declining policy, unlike the other RRC rows; and the Kumiho comparison mixes reported and reproduced numbers.

  6. Cost is not accounted for. On LongMemEval, RRC uses about 4x the LLM calls of the baseline (191,863 vs. 47,397) for no significant gain.

Questions for authors
  1. What is RRC's gain over the baseline under the same answer policy and the same scoring (five-category average including adversarial)?
  2. How exactly is C* computed, and what values of κ and s_d are used? Which routing path and answer policy were actually run at 1.5B?
  3. What is the quiet/noisy threshold, and how sensitive is q̂ to node-name normalization?
  4. Given that vote budget and majority voting barely affect results, what does the noise path actually contribute?
1 comment

We thank the reviewer for the detailed and constructive review.

W1 and W5. We agree that the main table mixes answer contracts and retrieval depths, which makes the comparison hard to read. The revision regroups the main table by answer contract, separates the retrieval protocols, and adds a crossed and pairwise analysis of contract and retrieval depth in which every cell is compared with the baseline.

W6. Extraction runs once per conversation, and serving a query reuses the stored graph, so the online cost is a single retrieval plus one reader call. The revision moves the full cost breakdown to an appendix.

Q1. We will report the matched-contract gains under the five-category scoring in the revised analysis section.

We thank the reviewer again for the close reading.

Reviewer_Ut5SOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a way to diagnose why LLM agent memory leads to failure. Authors claim that there are two parts: (1) Extraction Noise, and (2) Representation Loss. (1) EN means the compressed form is capable of storing that important piece of information and the loss is in the noisy extraction process, (2) RL means the compressed form representation under certain budget is not capable of storing/representing that information. Authors deviate (1) and (2) by check something like pass@infinity, Var[extracted information], etc.

Authors propose different solutions for different failure. For example, for high-variance + model capable enough they propose BoN-like structure; for low-variance+model capable enough they propose to enlarge the search range. For not capable enough model they propose to admit "not mentioned".

Critical strengths

(1) The authors propose a valid and interesting framework for understanding why and how memory compression fails. The theoretical analysis is valid and clearly represents their claim and proposed solution.

(2) Experiments show the validness of their method.

Critical weaknesses

(1) My main concern is that many SOTA agent models not only use one "compression" strategy. For example, it is said in newest codex versions, past conversation history (before compression) is exposed to agents as querable histories, where agents might search past memory for more information. If authors could include this in their analysis that would be very great.

(2) Some of the experiments cannot fully convince readers of the proposed advantage, there might be other noises. For example different prompts have big impact on scores (e.g. 7B model from 37.22 to 43.22, comparing gentle v.s. plain F1 score for LoCoMo).

Questions for authors

Please see weakness (1).

2 comments

We thank the reviewer for the thoughtful and detailed comments.

W1. We agree that the memory tiering deserves fuller treatment. The revision adds a dedicated discussion of what is stored at each tier and of when the raw history is queried directly.

W2. The reviewer is right that the answer contract changes the score and that this is part of the method, not an experimental detail. In proposed method, the policy choice is fixed before deployment, and the diagnosis selects the contract, so no prompt library is searched at run time. The revision states this in the method section and reports the policy contrast per capability band.

We thank the reviewer for helping us improve the presentation and the method description.

I'd like to thank the authors for their updates.

Overall I think the proposed theory analysis is very interesting, and might inspire further thoughts.

I'd encouragen authors to carry out more analysis and visualization about these theories in later versions (either camera ready or later versions).

Overall I'd increase my score to 3 weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.