- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates when brain-inspired components improve agent memory through a matched-substitution evaluation protocol. The authors systematically examine three aspects of memory systems: storage substrates, memory dynamics, and retrieval routing, comparing each component against functional alternatives under controlled conditions. Experiments across synthetic workloads and multiple existing memory systems show that dense storage often outperforms sparse distributed memory, temporal decay helps under specific conditions, and graph retrieval improves multi-hop reasoning at additional computational cost. The paper provides empirical insights into the effectiveness and limitations of different memory mechanisms.
Critical strengths
-
The paper systematically evaluates storage, memory dynamics, and retrieval routing across multiple memory systems and benchmarks. The extensive experiments and controlled comparisons provide valuable insights into the effectiveness of different memory mechanisms.
-
The authors employ careful experimental controls, statistical tests, and robustness analyses to support their findings. They also explicitly report inconclusive results and experimental limitations, which strengthens the credibility and reproducibility of the study.
Critical weaknesses
-
The proposed matched-substitution protocol mainly formalizes established evaluation practices, such as controlled component replacement, matched resource budgets, and mechanism activation checks. While the systematic experiments are valuable, the methodological contribution itself appears relatively incremental.
-
The paper provides extensive analyses of storage substrates, memory dynamics, and retrieval routing, but the findings remain largely independent. A more unified analysis of how these components interact, along with clearer practical design recommendations, would strengthen the overall contribution and significance of the work.
Questions for authors
The paper evaluates storage substrates, memory dynamics, and retrieval routing largely independently. Have the authors considered how these components interact in a unified memory system? For example, could combining dense storage, temporal decay, and graph retrieval lead to additional gains or introduce unexpected trade-offs?
4 comments
W2: The component findings lack a unified analysis and practical design guidance
Response: We agree that studying the three components independently cannot establish how their effects combine. Our approach first identifies component-level effects under controlled conditions, then examines whether these conclusions persist when components interact. The original shared-state and closed-loop experiments already investigate substrate–feedback interactions. We have now added a direct decay–routing combination test in a unified memory system (Q1).
We have also clarified the practical guidance by connecting each design decision to the evidence needed before adopting it:
| Design decision | Evidence needed for adoption |
|---|---|
| Select a storage substrate | Compare functioning compact dense controls under explicit byte accounting, then check whether the result survives each arm's own feedback. |
| Enable temporal decay | Verify that the operative clock and trace states change ranking; measure answer benefit on the target workload. |
| Choose reinforcement | Compare all-hit, selective, and no reinforcement using both stale-rule and current-rule retrieval. |
| Add graph routing or combine mechanisms | Measure the relevant policy increment and its cost, then test the combination and interaction directly rather than adding gains from separate workloads. |
The new answer-free-cue experiment further supports the reinforcement recommendation. Across 20 seeds and both dense and hybrid modes, it tests alternative reuse and probing cues without supplying either answer. At day 730, all-hit reinforcement returns stale rules in 100% of dense probes, versus 0% with top-1 or no reinforcement. The hybrid current/current condition also improves with top-1 relative to all-hit (Appendix S.1). This supports selective reinforcement as a mitigation for co-reinforcement, without establishing superiority over no reinforcement.
The new routing comparison similarly identifies a supported +5.89-point F1 gain [3.48, 8.35] from the implemented filter-and-fallback policy over raw-link PPR, although it requires substantially more inference resources (Appendix S.2).
Together, these findings provide a unified design principle: component choices should be evaluated through their individual contributions, interactions, and downstream quality–cost trade-offs, rather than assuming that gains observed separately will accumulate in a combined system.
Q1. Could combining dense storage, temporal decay, and graph retrieval produce additional gains or unexpected trade-offs?
We have completed the proposed temporal 2 × 2 experiment, holding dense storage fixed and crossing decay off/on with dense/graph retrieval. It generates 2,000 fresh answers on the same 500 RealMem queries across ten simulated personas. This directly tests whether combining the two mechanisms provides additional answer-quality gains or introduces trade-offs.
All four arms use the same Qwen3.5-397B-A17B-FP8 reader, float32 storage, and limits of five chunks, 6,000 evidence tokens, and 1,500 current-conversation tokens. Each query accesses only earlier sessions, with the graph reconstructed from the visible prefix. Decay applies the fixed MAM age factor to the final retrieval score; reconsolidation and dormancy are disabled to isolate the decay–routing combination rather than the full MAM controller.
| Policy | Normalized rubric score | Read tokens per query |
|---|---|---|
| Dense without decay D0G0 | 46.07 | 5,025.50 |
| Dense with decay D1G0 | 43.20 | 4,996.75 |
| Graph without decay D0G1 | 45.87 | 8,137.04 |
| Graph with decay D1G1 | 45.07 | 8,116.63 |
Each arm contains 500 answers. Blind grading uses the RealMem 0–3 rubric with a provider-reported DeepSeek-V3.2 judge from a different model family. Scores are divided by three and expressed on a 0–100 scale; they are not accuracy.
The two planned primary contrasts use 20,000 paired-persona bootstrap draws, with Bonferroni-adjusted 97.5% intervals:
| Primary comparison | Difference in points | Adjusted interval |
|---|---|---|
| Combined minus dense baseline | −1.00 | [−3.53, 1.20] |
| Decay by routing interaction | +2.07 | [−0.13, 5.27] |
The interaction is (D1G1 − D0G1) − (D1G0 − D0G0).
Neither comparison establishes a positive combined benefit or synergy; the intervals also do not establish equivalence. Decay changes the ordered top-five chunk lists on 372/500 dense queries and 297/500 graph queries. Thus, the mechanism is active, but its effects on retrieval do not necessarily translate into better final answers.
The combination also introduces a concrete resource trade-off: graph with decay uses 1.615× the baseline's read tokens. These include prompts, completions, and the graph policy's standalone filtering requirement, but exclude upfront graph construction and encoding. The execution shared physical filter calls across decay conditions; standalone accounting assigns filtering cost to each graph arm. All 218 valid-empty-filter dense fallbacks remain included.
All eight output-capped answers remain in the primary analysis. Assigning them zero quality gives a combined effect of −1.20 [−3.73, 1.00] and interaction of +2.40 [0.13, 5.53]. Since capped-answer counts differ across arms, this sensitivity result does not replace the primary finding.
The earlier RealMem storage/reinforcement experiment remains distinct: dense-top1 minus hybrid-all is +2.33 points [−2.13, 7.07], with an unresolved interaction of −0.87 [−4.40, 3.07]. We do not pool it with the new decay/routing experiment.
The new study uses ten previously exposed simulated personas and model-based grading, so broader population-level transfer remains unresolved. Nevertheless, it directly answers the reviewer's question: combining active memory mechanisms does not necessarily produce additional answer-quality gains and can introduce substantial resource costs.
These findings reinforce why component attribution and interaction analysis are both necessary. Individual component effects provide useful diagnoses, but their combination and practical utility must be evaluated directly rather than inferred from separate experiments.
We hope the clarified methodological motivation, completed interaction experiment, and practical design guidance address the concerns about the paper's contribution and overall significance.
Authors have addressed my concerns and I will keep the rating
Thank you for recognizing the systematic evaluation and careful experimental controls. We appreciate the concerns about methodological novelty and interactions among memory components. We have clarified the motivation behind our evaluation design and added a cross-component experiment to address these questions.
W1: The methodological contribution appears incremental
Response: We agree that controlled replacement, resource matching, and activation checks are established techniques. Our contribution lies in designing and organizing these techniques to address a specific problem: how to determine whether a brain-inspired memory component actually provides an advantage over a functioning alternative, rather than merely whether the system depends on it.
This task introduces several challenges that conventional removal ablations do not resolve. A replacement must serve the same function under comparable resource constraints; the mechanism must actually affect the evaluated path; and when retrieval changes subsequent memory states, direct readout effects must be distinguished from feedback-mediated effects. Our matched-substitution protocol addresses these requirements through a coordinated experimental design, rather than treating each control independently.
The experiments demonstrate why these design choices are necessary.
Representation can reverse the attributed advantage. At a previously SDM-favorable operating point, compact dense storage under the same operational-byte accounting changes the contrast to a 7.01-point SDM deficit (paired-seed 90% interval [−7.79, −6.20]). This identifies a capacity explanation for the original apparent advantage, rather than simply reporting a performance difference (Section 3.2; Figure 3(a)).
Internal updates do not necessarily make a mechanism observable. Proposition 2 establishes identical dense readouts under a fixed clock and saturated scoring, even when reinforcement updates internal weights. This identity holds on all 1,200 paired rows. Binding the operative clock to corpus time then increases strict update wins from 44/78 to 60/78, with no reversals. Thus, activation checks establish whether an experiment can actually measure the intended mechanism (Section 4.1; Figure 3(b)).
Readout and feedback require different interventions. Constructive projection isolates the direct substrate effect along a shared controller trajectory, while the separate 1,280-stream closed-loop experiment tests whether the ordering persists when each arm updates from its own retrievals (Section 3.1; Table S10).
The novelty is therefore not in the individual evaluation techniques, but in their task-driven design as a coherent attribution protocol, together with the empirical findings it enables. We believe both the methodology and the resulting evidence should be considered when assessing the contribution.