- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that failures in LLM-agent memory construction fall into two regimes: (i) random sampling noise, which can be eliminated by repeatedly sampling from the same history and taking a majority vote; and (ii) representational loss caused by a capability-limited backbone, which cannot be eliminated and can only be handled by abstention. Based on this distinction, the authors propose RRC, a training-free routing layer that runs extractions over the same window, computes inter-run disagreement (the average pairwise Jaccard distance between extracted node sets) as an estimator , and, together with a capability indicator ( parameter count) and a question reasoning-depth coordinate , routes among four configurations (add voting, deeper retrieval, decline policy, and default). The paper provides a sketch formal model and a Fano-type argument establishing the existence of a capability threshold and the property that the estimator tracks only noise, not bias. Experiments are conducted on LoCoMo and LongMemEval.
Critical strengths
- S1. The core idea is clean and well motivated. Treating the stochasticity of the construction pipeline itself as a free diagnostic signal, and moving the consistency signal from the answer level (self-consistency, semantic entropy) to the construction level in order to diagnose Regime I, is a reasonable and non-obvious move.
- S2. The formalization and proofs are self-consistent. Propositions 1–6 are standard uses of tools such as Hoeffding/Fano/U-statistics and capture the correct invariants; the cross-capability gain/loss patterns are qualitatively consistent with the two-regime story on the numbers reported by the authors.
Critical weaknesses
- W1. Within-group rankings in the main experiments do not control for backbone, and the “bold is best, underline second best per group” convention actually presents cross-backbone rankings. In the Small group (1.5B–3B), GAM is run on Llama-3.2-3B (36.27), Synapse is run on Qwen-1.5B (35.9) and Qwen-3B (36.6), and the A-Mem/MemoryBank/MemoryOS rows are cross-backbone reruns of Synapse on Qwen-1.5B/Qwen-3B (B.1), whereas the authors’ own Baseline/RRC rows are run on Qwen2.5-3B and Llama-3.2-3B. The Medium group mixes Qwen2.5-7B, LLaMA-3.1-8B, and Qwen2.5-14B; the Large group mixes a single Long-context baseline row (Llama-2-70B-Chat, 17.9) into Qwen2.5-72B rows; and in the API group all competitors are GPT-4o/GPT-4o-mini while this method uses DeepSeek-V3. This directly conflicts with the statement in §4.1 that “Table 1 compares against published memory systems at matched backbones.” The headline in the abstract and conclusion that exceeds Kumiho (57.8/58.5 vs. 56.5) is a cross-backbone comparison of DeepSeek-V3 against GPT-4o and cannot support conclusions about method superiority.
- W2. The same column of the main table mixes incomparable scoring protocols. The authors’ recipe rows average four non-adversarial categories, while the headline rows average the official five categories (Table 1 caption); GAM and Synapse published values are four-category averages excluding adversarial (B.1); the remaining published values are self-reported overall token F1 from their respective papers; Kumiho’s 56.5 includes an adversarial score of 97.5 (its pinned reproducible value is only 53.1, B.1); and Memory-R1’s LongMemEval numbers (46.70/45.20, †) use its self-selected subset and scoring protocol but are placed in the same column as the authors’ weighted micro-F1 and included in within-group rankings. On the one hand, the authors state that “Judge scores are not comparable across judges, and we do not rank against leaderboard entries scored by other judges” (B.1), and that LongMemEval judge scores are “not comparable across systems” (§4.2); on the other hand, Table 1 presents exactly such a mixed ranking.
- W3. The core paired comparison is asymmetric in answer policy, and other configuration dimensions are also asymmetric. §4.1 (l.366–367) states that the baseline uses “the gentle answer policy” and that “RRC differs only in vote, estimation, and routing,” but the Table 10 footnote reveals that “the RRC recipe rows run the plain answer policy,” and Table 12 shows that, on the same RRC framework, gentle is clearly lower than plain (3B: 40.26 vs. 43.39; 7B: 37.22 vs. 43.22). Therefore, the paired gains reported in the main text (e.g., 3B +3.79, 7B +5.29) likely come mainly from the answer-policy mismatch rather than from the RRC mechanism itself. The largest gain, 8B +9.93, further stacks a bge-base retriever upgrade applied only to RRC rows (Table 10 footnote, C.1). Moreover, each backbone reports its own best arm: 3B/7B choose plain, 8B chooses the bge upgrade (46.68), 14B chooses the grounded policy (45.05), Llama-3.2-3B chooses grounded (on this backbone the plain gain is only +0.08, C.1), and the API headline chooses gentle+top-75 (l.380–381); by contrast, each compared published system has only one fixed configuration. Grounded is not among the four routing paths in Table 4, and the bge-base upgrade is not a routing parameter; both conflict with “differs only in vote, estimation, and routing.”
- W4. The paper’s own ablations undermine its core mechanism. The core prescription for Regime I—increasing the number of extraction votes—has no measurable effect: yields 40.59/40.26/40.09, a range of 0.50 within the noise; the paper itself admits that “the merged store is vote-neutral; votes do not denoise it directly” (§4.4). Table 2 further shows that replacing majority voting with union merging yields a higher 3B result (43.75 vs. 43.39) and a roughly equal 7B result (43.07 vs. 43.22). The only ablation that produces a large difference is the decline clause (replacing it with verify-then-decline collapses to 30.12/30.33), but that is an answer policy, not the estimator or routing. The paper also does not provide an ablation of the router itself (e.g., forcing routing to each path). Therefore, the mechanism claimed as the core contribution—the estimator plus routing—is not shown by the ablations to contribute any part of the gain; combined with W3, the observed gains are more likely explained by policy switching and the retriever upgrade.
Questions for authors
- Q1. Can the authors rerun the main baselines under a unified harness and a unified backbone, and provide a truly comparable main table?
- Q2. Please add results for {baseline, RRC} × {plain, gentle} on at least two fixed backbones, and report the actual gains under a matched answer policy.
- Q3. Please eliminate the asymmetries across configuration dimensions in the core comparison: do not perform post-hoc arm selection per backbone, do not upgrade the retriever on only one side, and report corrected gains. In addition, please add a routing ablation (forcing each path vs. a fixed configuration) to separate the contributions of the estimator and the routing mechanism itself.
We thank the reviewer for the careful review.
W2. As noted in our reply to Reviewer 2, the revision regroups the main table by answer contract and separates the retrieval protocols.
W3. We agree that the retrieval-side comparison should be run under a single harness. The revision reruns all single-pass baselines with the same reader, encoder, list depth, and answer contract, and reports paired differences with confidence intervals.
W4. The ablations are intended to isolate the configuration layer. The revision reorders them to make this role explicit and adds the routing ablation under the unified harness.
We thank the reviewer for the comments, which have directly shaped the revision plan.