acceptodds
Desk rejected at ICLR 2027

When Do Brain-Inspired Components Improve Agent Memory?

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-term agent memory increasingly uses brain-inspired storage, forgetting, reinforcement, and retrieval, yet system-level gains leave the advantage of each mechanism unclear. Removing a component tests system dependence; it does not establish superiority over a functioning alternative. We formulate component attribution as a comparison defined by the replacement, measured endpoint, and state or resource constraint. Our matched-substitution protocol applies this formulation to storage substrates, memory dynamics, and retrieval routing, checks mechanism activation, and separates shared-state readout contrasts from closed-loop comparisons that allow retrieval feedback. These controls change what the observations support. Hybrid sparse distributed memory lowers answer accuracy by about 14.8 percentage points relative to dense storage on frozen synthetic streams, while favorable readout cells disappear against compact equal-byte dense controls. Temporal decay helps in tested regimes with an advancing clock and distinguishable trace states, whereas joint reinforcement can revive stale rules. Chronological answer replay leaves the joint benefit of dense readout and restricted reinforcement unresolved. Prospective graph retrieval improves bridge-question F1 by 9.24 points under a read constraint; a separate lifecycle study measures substantial construction and maintenance costs. The resulting design guidance is conditional: compare compact storage alternatives, test when temporal updates affect ranking, and evaluate routing quality alongside cost. Reliable attribution requires functioning substitutions within a stated operating regime, beyond removal effects alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-term agent memory, Brain-inspired memory, Component attribution, Controlled evaluation, Sparse distributed memory, Memory dynamics, Retrieval routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_umBNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when brain-inspired components improve agent memory through a matched-substitution evaluation protocol. The authors systematically examine three aspects of memory systems: storage substrates, memory dynamics, and retrieval routing, comparing each component against functional alternatives under controlled conditions. Experiments across synthetic workloads and multiple existing memory systems show that dense storage often outperforms sparse distributed memory, temporal decay helps under specific conditions, and graph retrieval improves multi-hop reasoning at additional computational cost. The paper provides empirical insights into the effectiveness and limitations of different memory mechanisms.

Critical strengths
  1. The paper systematically evaluates storage, memory dynamics, and retrieval routing across multiple memory systems and benchmarks. The extensive experiments and controlled comparisons provide valuable insights into the effectiveness of different memory mechanisms.

  2. The authors employ careful experimental controls, statistical tests, and robustness analyses to support their findings. They also explicitly report inconclusive results and experimental limitations, which strengthens the credibility and reproducibility of the study.

Critical weaknesses
  1. The proposed matched-substitution protocol mainly formalizes established evaluation practices, such as controlled component replacement, matched resource budgets, and mechanism activation checks. While the systematic experiments are valuable, the methodological contribution itself appears relatively incremental.

  2. The paper provides extensive analyses of storage substrates, memory dynamics, and retrieval routing, but the findings remain largely independent. A more unified analysis of how these components interact, along with clearer practical design recommendations, would strengthen the overall contribution and significance of the work.

Questions for authors

The paper evaluates storage substrates, memory dynamics, and retrieval routing largely independently. Have the authors considered how these components interact in a unified memory system? For example, could combining dense storage, temporal decay, and graph retrieval lead to additional gains or introduce unexpected trade-offs?

4 comments

Thank you for recognizing the systematic evaluation and careful experimental controls. We appreciate the concerns about methodological novelty and interactions among memory components. We have clarified the motivation behind our evaluation design and added a cross-component experiment to address these questions.

W1: The methodological contribution appears incremental

Response: We agree that controlled replacement, resource matching, and activation checks are established techniques. Our contribution lies in designing and organizing these techniques to address a specific problem: how to determine whether a brain-inspired memory component actually provides an advantage over a functioning alternative, rather than merely whether the system depends on it.

This task introduces several challenges that conventional removal ablations do not resolve. A replacement must serve the same function under comparable resource constraints; the mechanism must actually affect the evaluated path; and when retrieval changes subsequent memory states, direct readout effects must be distinguished from feedback-mediated effects. Our matched-substitution protocol addresses these requirements through a coordinated experimental design, rather than treating each control independently.

The experiments demonstrate why these design choices are necessary.

Representation can reverse the attributed advantage. At a previously SDM-favorable operating point, compact dense storage under the same operational-byte accounting changes the contrast to a 7.01-point SDM deficit (paired-seed 90% interval [−7.79, −6.20]). This identifies a capacity explanation for the original apparent advantage, rather than simply reporting a performance difference (Section 3.2; Figure 3(a)).

Internal updates do not necessarily make a mechanism observable. Proposition 2 establishes identical dense readouts under a fixed clock and saturated scoring, even when reinforcement updates internal weights. This identity holds on all 1,200 paired rows. Binding the operative clock to corpus time then increases strict update wins from 44/78 to 60/78, with no reversals. Thus, activation checks establish whether an experiment can actually measure the intended mechanism (Section 4.1; Figure 3(b)).

Readout and feedback require different interventions. Constructive projection isolates the direct substrate effect along a shared controller trajectory, while the separate 1,280-stream closed-loop experiment tests whether the ordering persists when each arm updates from its own retrievals (Section 3.1; Table S10).

The novelty is therefore not in the individual evaluation techniques, but in their task-driven design as a coherent attribution protocol, together with the empirical findings it enables. We believe both the methodology and the resulting evidence should be considered when assessing the contribution.

W2: The component findings lack a unified analysis and practical design guidance

Response: We agree that studying the three components independently cannot establish how their effects combine. Our approach first identifies component-level effects under controlled conditions, then examines whether these conclusions persist when components interact. The original shared-state and closed-loop experiments already investigate substrate–feedback interactions. We have now added a direct decay–routing combination test in a unified memory system (Q1).

We have also clarified the practical guidance by connecting each design decision to the evidence needed before adopting it:

Design decisionEvidence needed for adoption
Select a storage substrateCompare functioning compact dense controls under explicit byte accounting, then check whether the result survives each arm's own feedback.
Enable temporal decayVerify that the operative clock and trace states change ranking; measure answer benefit on the target workload.
Choose reinforcementCompare all-hit, selective, and no reinforcement using both stale-rule and current-rule retrieval.
Add graph routing or combine mechanismsMeasure the relevant policy increment and its cost, then test the combination and interaction directly rather than adding gains from separate workloads.

The new answer-free-cue experiment further supports the reinforcement recommendation. Across 20 seeds and both dense and hybrid modes, it tests alternative reuse and probing cues without supplying either answer. At day 730, all-hit reinforcement returns stale rules in 100% of dense probes, versus 0% with top-1 or no reinforcement. The hybrid current/current condition also improves with top-1 relative to all-hit (Appendix S.1). This supports selective reinforcement as a mitigation for co-reinforcement, without establishing superiority over no reinforcement.

The new routing comparison similarly identifies a supported +5.89-point F1 gain [3.48, 8.35] from the implemented filter-and-fallback policy over raw-link PPR, although it requires substantially more inference resources (Appendix S.2).

Together, these findings provide a unified design principle: component choices should be evaluated through their individual contributions, interactions, and downstream quality–cost trade-offs, rather than assuming that gains observed separately will accumulate in a combined system.

Q1. Could combining dense storage, temporal decay, and graph retrieval produce additional gains or unexpected trade-offs?

We have completed the proposed temporal 2 × 2 experiment, holding dense storage fixed and crossing decay off/on with dense/graph retrieval. It generates 2,000 fresh answers on the same 500 RealMem queries across ten simulated personas. This directly tests whether combining the two mechanisms provides additional answer-quality gains or introduces trade-offs.

All four arms use the same Qwen3.5-397B-A17B-FP8 reader, float32 storage, and limits of five chunks, 6,000 evidence tokens, and 1,500 current-conversation tokens. Each query accesses only earlier sessions, with the graph reconstructed from the visible prefix. Decay applies the fixed MAM age factor to the final retrieval score; reconsolidation and dormancy are disabled to isolate the decay–routing combination rather than the full MAM controller.

PolicyNormalized rubric scoreRead tokens per query
Dense without decay D0G046.075,025.50
Dense with decay D1G043.204,996.75
Graph without decay D0G145.878,137.04
Graph with decay D1G145.078,116.63

Each arm contains 500 answers. Blind grading uses the RealMem 0–3 rubric with a provider-reported DeepSeek-V3.2 judge from a different model family. Scores are divided by three and expressed on a 0–100 scale; they are not accuracy.

The two planned primary contrasts use 20,000 paired-persona bootstrap draws, with Bonferroni-adjusted 97.5% intervals:

Primary comparisonDifference in pointsAdjusted interval
Combined minus dense baseline−1.00[−3.53, 1.20]
Decay by routing interaction+2.07[−0.13, 5.27]

The interaction is (D1G1 − D0G1) − (D1G0 − D0G0).

Neither comparison establishes a positive combined benefit or synergy; the intervals also do not establish equivalence. Decay changes the ordered top-five chunk lists on 372/500 dense queries and 297/500 graph queries. Thus, the mechanism is active, but its effects on retrieval do not necessarily translate into better final answers.

The combination also introduces a concrete resource trade-off: graph with decay uses 1.615× the baseline's read tokens. These include prompts, completions, and the graph policy's standalone filtering requirement, but exclude upfront graph construction and encoding. The execution shared physical filter calls across decay conditions; standalone accounting assigns filtering cost to each graph arm. All 218 valid-empty-filter dense fallbacks remain included.

All eight output-capped answers remain in the primary analysis. Assigning them zero quality gives a combined effect of −1.20 [−3.73, 1.00] and interaction of +2.40 [0.13, 5.53]. Since capped-answer counts differ across arms, this sensitivity result does not replace the primary finding.

The earlier RealMem storage/reinforcement experiment remains distinct: dense-top1 minus hybrid-all is +2.33 points [−2.13, 7.07], with an unresolved interaction of −0.87 [−4.40, 3.07]. We do not pool it with the new decay/routing experiment.

The new study uses ten previously exposed simulated personas and model-based grading, so broader population-level transfer remains unresolved. Nevertheless, it directly answers the reviewer's question: combining active memory mechanisms does not necessarily produce additional answer-quality gains and can introduce substantial resource costs.

These findings reinforce why component attribution and interaction analysis are both necessary. Individual component effects provide useful diagnoses, but their combination and practical utility must be evaluated directly rather than inferred from separate experiments.

We hope the clarified methodological motivation, completed interaction experiment, and practical design guidance address the concerns about the paper's contribution and overall significance.

Authors have addressed my concerns and I will keep the rating

Reviewer_eM9uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the usual way of crediting a brain-inspired component in agent memory, removing it and measuring the drop, only shows that the system depends on that component. It does not show that the brain-inspired version is better than a plain alternative doing the same job. The authors instead replace each component with a functioning alternative under a stated resource constraint, and check that the mechanism is actually active on the path being evaluated. The protocol is applied to storage, memory dynamics and retrieval routing, mostly inside the authors' own system (MAM), with further tests on MemoryBank, HippoRAG 2, MG2-RAG and HippoMM. Sparse distributed memory (SDM) loses to dense storage at equal bytes, and its few favorable cells disappear once the dense vectors are kept in a compact format. In the tested fixtures, decay makes no difference unless the retention clock actually moves, and reinforcing an old rule together with its correction can bring the old rule back. Graph retrieval helps on multi-hop bridge questions, at a much higher build and read cost. The chronological answer replay on RealMem is inconclusive.

Critical strengths

The core point is simple and, I think, right: showing that a system gets worse without a component is not the same as showing that the component beats a functioning alternative. The paper has a good example of why this matters. Removing feedback from MAM costs 67 to 77 points, but with the clock held fixed and dense readout, append-only storage returns exactly the same results as the full controller on every paired row, so none of that loss can be credited to the retention mechanism. This is the part of the paper I would want people working on agent memory to read.

The activation checks also turned up real problems in other people's systems. I looked at the public MemoryBank code to check one of them, and it is as the paper describes. enable_forget_mechanism defaults to False, and the retention function returns math.exp(-t / 5*S), which Python evaluates with SS in the numerator, so memories with higher strength are forgotten faster, the opposite of what the function's own docstring says. Findings like this are worth having whatever one thinks of the paper's main conclusions.

Critical weaknesses

My main reservation is how much the study tells us about brain-inspired components themselves. The abstract is careful to present its guidance as conditional, but the title asks a general question, and what is tested are specific implementations of three components, with the storage and dynamics evidence coming mostly from the authors' own system and a synthetic generator they wrote. Several of the headline results also come close to following from how things are set up. Stripped down, the storage comparison is a Kanerva-style SDM over random-projection hypervectors against dense rows, in a configuration where a bit-packed dense row costs 156 bytes and the SDM substrate costs 1.3 MB. The paper itself notes that binary dense storage is then never capacity-constrained, so it is no surprise that SDM's favorable cells disappear once compact dense formats are allowed. The dynamics results are similar. At a fixed clock the decay factor is exactly one, and the score factor stops changing once a trace's weight reaches one, so the identity between the full controller and append-only storage follows from these two facts, as the paper's own Proposition 2 shows. Under reuse, the stale rule wins on dense storage because co-recall drives both traces to the same weight (Theorem 1), and the stale-rule fixture was built post hoc, with a cue that quotes the stale rule word for word. A sensitivity sweep finds the same failure in almost every cell on dense storage, but it is weaker on the hybrid substrate and with image pairs. These constructed cases are useful demonstrations of how an evaluation can mislead, and I credit them as such, but they say little about when brain-inspired memory actually helps, and the chronological answer replay on RealMem does not show a benefit in any contrast. The title should be narrowed to what was actually tested.

My second concern is presentation, and it limits how far I could check the claims. The main text is mostly a set of pointers into a much longer appendix, and I could not follow Sections 3 to 5 without reading the appendix. The zombie rate, the unit of Table 3, is never formally defined, and neither is the per-item term in the byte budget. The readout plots in Figs. 2 and 3(a) also show no uncertainty. Much of the appendix is process record: commit hashes, freeze times, request and token counts, amendment histories. Much of this does not help a reader judge the results, and by the paper's own account in Appendix C the manifests establish byte integrity, not when experiments were run, so the "frozen before outcomes" claims cannot be checked from the paper itself. Some numbers also do not reconcile. This looks like a reporting problem rather than a threat to the large gaps, but it should be fixed. In Table S3(b) the pure-SDM deltas do not match the listed accuracies; subtracting the listed accuracies gives −35.00-35.00 points where the table reports −35.42-35.42. They may be paired on a subset of the rows, but the paper does not say.

Questions for authors
  1. Clarification: Which deployed agent-memory systems are byte-constrained in the way the storage comparison assumes, and how representative are the tested byte budgets and loads of them?
  2. Clarification: Can you reconcile the Table S3(b) deltas with the listed accuracies, and the −3.91-3.91 point Full MAM minus dense_hot difference for Qwen in Table S3(a) with the −4.16-4.16 points implied by the Table S9(a) accuracies? Which answer model and rows does Table S9(a) use?
  3. Suggestion: Narrow the title to the tested mechanisms and operating regime, define the zombie rate, the operational byte budget and its per-item term in the main text, and trade some of the process documentation for space to explain the results.
2 comments

Thank you for recognizing the value of distinguishing component dependence from superiority over a functioning replacement, and for independently checking our MemoryBank findings. We appreciate the concerns about generality, experimental design, and presentation. We have added targeted experiments and revised the claims and reporting accordingly.

W1. “Limited generality and potentially constructed findings.”

Response: We agree that several findings are specific to the tested implementations and operating conditions. However, identifying these conditions is central to our study: a component's observed effect should not automatically be attributed to its brain-inspired mechanism or generalized beyond the evaluated regime.

For storage, the original budget makes compact dense favorable by construction. We explicitly acknowledge this and further test externally fixed capacity constraints (Q1). For dynamics, the fixed-clock identity indeed follows from Proposition 2. We use it as an activation diagnosis, not evidence that decay is generally ineffective. Importantly, advancing the clock on 78 LongMemEval update questions increases current-evidence wins from 44 to 60, with 16 improvements and no reversals.

We also addressed the concern that the stale-rule fixture favors the old answer through its query. The original fixture was inaccurately described in Appendix M: its cue uses tenant/topic identifiers rather than quoting the stale answer. As the historical driver is not fully recovered, we performed a new controlled experiment using answer-free cues. Across all four wording combinations, dense retrieval at day 730 returns stale rules in 100% of all-hit reinforcement cases, versus 0% with top-1 or no reinforcement. Under hybrid retrieval, top-1 reduces stale returns by 33.33 pp [26.25, 40.83] relative to all-hit (Appendix S.1). This supports the co-reinforcement failure diagnosis without establishing its prevalence in deployed systems.

We agree that broader answer-level benefits remain uncertain. New RealMem and LongMemEval experiments do not establish positive effects of the tested temporal interventions (Q3). Conversely, a new HotpotQA routing comparison identifies a supported +5.89 pp F1 gain from the implemented filter-and-fallback policy, albeit at substantial additional cost.

These results support conditional attribution to implemented mechanisms, not general claims that brain-inspired memory components are effective or ineffective across all systems.

W2: Presentation and verifiability

Response**:** We agree that the original presentation placed too much burden on the appendix. We have revised Sections 3–5 to explain the comparisons, operating conditions, and interpretation of the results more directly, rather than relying primarily on appendix references. Definitions and uncertainty displays are addressed in Q3, and the numerical discrepancies are corrected in Q2.

We also clarify the evidence behind our protocol records. Internal freeze records document the reported experimental procedures, while hashes establish artifact integrity, not independently verifiable chronology or public preregistration. We now state this distinction explicitly in the Reproducibility Statement and Limitations, rather than treating the records as stronger evidence than they provide.

The detailed execution histories remain available for reproducibility, but are no longer relied upon to explain the scientific conclusions.

Q1: Are the byte budgets representative of deployed agent-memory systems?

Response**:** We agree that our original operational-byte budget is not a measured typical deployment constraint. The SDM implementation uses 1,310,720 fixed substrate bytes plus 1,052 bytes per trace, whereas packed dense uses 156 bytes per trace. Thus, packed dense is never capacity-constrained under this particular accounting. We now make this property explicit.

An audit of 7,031 memory-search calls and 4,627 trace rows from two agent instances provides observed workload scale, but does not establish representative production budgets or permit controlled outcome replay.

To address the comparison-fairness concern, we added an externally fixed-cap study with five implementations, nine persistent-buffer caps, and three seeds on 9,811 HotpotQA passages (Appendix T). At 1 MiB, packed dense retains only 6,721 passages and is genuinely capacity-constrained. At 8 MiB, both packed SDM and packed dense retain all passages, yet supporting-document fraction@5 is 0.05% versus 78.50%. All six planned SDM-minus-packed-dense contrasts have multiplicity-adjusted intervals below zero.

This suggests that the tested readout deficit cannot be explained solely by capacity allocation. These are static binary-hypervector retrieval results, not end-to-end accuracy or whole-process RAM measurements. We also retain SDM's exploratory +7.08 pp advantage at 128× overload under the distinct equal-substrate-byte convention. The practical recommendation is therefore to compare functioning representations under the actual resource constraints, rather than assume our numerical budget applies universally.

Q2: Can the Table S3 and S9 discrepancies be reconciled?

Response**:** Yes. The reviewer correctly identified a reporting error: Table S3(b) mixed micro-averaged comparator accuracies with macro-averaged differences. We have corrected this without regenerating or excluding answers.

Table S9(a) uses Qwen3.5-397B-FP8. Accuracy and evidence recall use all 720 post-feedback rows per arm; latency uses all 1,200 rows. S9 reports micro averages, while S3 gives equal weight to 384 rule-within-setting groups (336 with two answers, 48 with one).

ContrastMicro differenceMacro difference
Full MAM − dense_hot−4.1667 pp−3.9063 pp
Pure SDM − Full MAM−35.0000 pp−35.4167 pp
Pure SDM − dense_hot−39.1667 pp−39.3229 pp

For example, Full MAM and dense_hot have micro accuracies of 80.5556% and 84.7222%, but macro accuracies of 81.7708% and 85.6771%. The latter explains the −3.91 pp difference in S3(a).

We now report consistently macro-weighted values in S3, label micro results separately, and clarify the row counts, weighting, and corresponding statistical tests in both captions. The correction does not change the saved outcomes or the substantive conclusions.

Q3: Narrowing the title and improving the main-text explanations

Response**:** We agree that the original title implied broader generality than the evidence supports. We have revised it to Evaluating Storage, Dynamics, and Routing in Brain-Inspired Agent Memory, making the focus on evaluated implementations clearer.

We have also addressed the specific presentation issues:

Zombie rate: Now explicitly defined as the fraction of all pair-specific probes returning the superseded rule at top 1, with misses and unrelated returns included in the denominator.

Byte accounting: The main text now defines the fixed substrate cost and each per-trace term, including why packed dense is unconstrained under the original operational budget.

Uncertainty: Figure 2 now displays pointwise 95% whole-seed intervals, and Figure 3(a) includes paired-difference 90% intervals. The constructed state identity remains distinguished from empirical readout effects.

Main-text clarity: Sections 3–5 now explain the essential results and their limitations more directly, while detailed protocol and execution records remain in the supplement.

We additionally evaluated downstream transfer rather than assuming retrieval effects imply answer benefits. The RealMem decay-by-routing factorial yields −1.00 pp [−3.53, 1.20] for the combined intervention. On 500 LongMemEval questions, decay changes top-five retrieval on 384 questions but yields −0.80 pp [−3.33, 1.62] in answer correctness (Appendices S and U). Neither establishes downstream superiority.

These findings further clarify the intended scope: a mechanism can be active and affect retrieval without establishing a final-answer benefit. Our revised presentation makes these distinctions explicit, while retaining both favorable and unfavorable results.

We hope these corrections and additional controls clarify the scientific contribution and address the concerns about generality, numerical consistency, and presentation.

Reviewer_B1qeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether brain-inspired memory components outperform functioning alternatives under explicit workload, state, and resource constraints. The authors use matched substitutions, activation checks, shared-state comparisons, and separate closed-loop experiments to study storage, memory dynamics, and retrieval routing. The results favor compact dense controls in the tested SDM comparisons, identify conditions under which temporal updates affect retrieval, and show graph-retrieval gains alongside separately measured costs and unresolved RealMem answer-level transfer.

Critical strengths
  1. The distinction between dependence on a component and superiority over a functioning replacement is valuable. Explicit state and resource controls, separate shared-state and closed-loop comparisons, held-out tuning, and independent implementation checks make the component analyses substantially more informative than removal-only ablations.

  2. The study provides useful conditional findings about clocks, reinforcement, storage representation, and graph retrieval. Reporting unresolved transfer results and policy-specific maintenance costs, rather than presenting uniformly favorable results, improves the interpretability of these findings.

Critical weaknesses
  1. The practical scope is narrower than the controlled diagnoses. Shared-state projection requires a state representable by both alternatives, limiting its applicability to systems that jointly rewrite traces and controller state. In Appendix O, the RealMem Dense-top1 minus hybrid-all contrast in normalized rubric score is +2.33 percentage points, with a 95% interval of [-2.13, +7.07] over ten simulated personas. The authors acknowledge this uncertainty, but broader answer-level transfer remains unresolved.

  2. The graph study leaves some attribution and deployment questions open. Full graph versus dense retrieval changes linking, filtering, graph access, and propagation together. The full-versus-one-step comparison isolates additional propagation but leaves its benefit uncertain (Appendix P). The separate lifecycle measurements use different update policies and do not establish a corresponding quality-adjusted cost advantage (Appendix Q).

Questions for authors
  1. Clarification: Which conclusions remain stable across the existing tuning, readout/address controls, and representation/budget sweeps? Please distinguish conclusions that require a common representable state from those supported by the separate closed-loop experiments.

  2. Additional validation: Beyond the full-versus-one-step comparison, could linking and filtering be isolated and evaluated together with lifecycle costs under corresponding execution conditions? If this is outside the present scope, which practical recommendations should remain conditional?

4 comments

Thank you for recognizing the value of distinguishing component dependence from superiority over a functioning replacement, and for raising important questions about robustness, attribution, and practical scope.

We agree with the two limitations highlighted in the review: 1) shared-state projection does not directly cover systems where retrieval also changes subsequent controller states, and 2) a full graph-versus-dense comparison cannot isolate the contributions of individual routing mechanisms or establish deployment efficiency. These distinctions are also central to our study. Our goal is not to establish that a particular memory design is universally superior, but to determine which conclusions can be attributed to which implemented mechanisms, under what conditions, and with what measured costs.

Presentation and reading burden. We also acknowledge your concern about the effort required to follow the argument. To make the main text more self-contained, we have defined the zombie rate and its denominator in Section 4.2, specified the operational-byte budget and per-trace terms in Section 3.2, and added uncertainty intervals to Figure 2 and Figure 3(a). Section 2 and Figure 1 present the comparison workflow. The nine-page main text retains its figures and tables and explains the operating conditions and implications alongside the results; detailed follow-up evidence is in Appendices S–U. These presentation revisions are intended to make the research questions, controlled comparisons and conclusions understandable before readers consult the detailed records.

Inspired by you valuable comments and suggestions, we have added experiments to address the reviewer's questions more directly and revised the paper accordingly.

W.1: "Which conclusions remain stable across the existing tuning, readout/address controls, and representation/budget sweeps? Please distinguish conclusions that require a common representable state from those supported by the separate closed-loop experiments."

Response: The main storage conclusion is stable across the tested readout implementations and closed-loop settings, but its scope depends on the replacement and budget convention. This distinction is important: robustness within a specified comparison does not imply that the same ordering must hold under every representation or resource constraint.

  • Robustness across existing controls. Across the primary 180-comparison grid, dense wins 178 comparisons, one meets the prespecified equivalence margin, and SDM wins one. Held-out activation-width tuning gives SDM 0.1803 versus equal-byte dense 0.7811; cleanup and alternative-address controls do not eliminate the deficit (Appendix E).

The representation sweep further clarifies the boundary of this conclusion. Across 48 cells, SDM wins 6/3/1/0 comparisons against float32/float16/int8/binary dense. At the originally SDM-favorable cell, replacing float32 dense with compact equal-operational-byte dense reverses the contrast to a 7.01-point SDM deficit. We also retain favorable SDM results, including the exploratory +7.08-point advantage at 128× overload under the different equal-substrate-byte convention (Appendix D).

Thus, the robust finding is not that SDM can never outperform dense storage. Rather, an apparent substrate advantage can disappear when the functioning alternative is given a more appropriate representation under the same budget. This is precisely why the replacement and resource constraint must be specified.

  • New validation under externally fixed budgets. To test whether these findings depend on the original budget accounting, we added a HotpotQA study crossing five implementations, nine persistent retrieval-buffer caps, and three retention seeds over 9,811 passages and 1,000 questions (135 planned configurations, including 12 infeasible settings). All arms use the same BGE-derived 1,024-bit codes.

Packed dense is genuinely capacity-constrained at 1 MiB, retaining 6,721 passages. At 8 MiB, both packed-decoder SDM and packed dense retain all 9,811 passages, yet their supporting-document fractions at five are 0.05% and 78.50%, respectively. All six planned SDM-minus-packed-dense intervals are negative after familywise adjustment. Matched-candidate direct decoding recovers the evidence, showing that retained-candidate count alone does not account for the native-readout deficit (Appendix T, pp. 44–45).

This extends the original diagnosis: under these fixed caps, the readout mechanism itself matters beyond storage capacity. The experiment measures static binary-hypervector retrieval, not hybrid verification or downstream answer accuracy. Persistent-buffer limits exclude separately measured process memory, and compression does not necessarily improve latency: at 64 MiB, packed dense takes 5.12 ms/query versus 1.08 ms for float32 bipolar dense.

  • What shared-state and closed-loop experiments establish. We agree that constructive projection requires a meaningful controller state representable by both alternatives. This is an identification condition, not an assumption that independently evolving memory systems share the same state.

Projection deliberately fixes the controller trajectory and severs retrieval-induced reconsolidation feedback. It therefore identifies the direct readout contrast along that trajectory, but cannot establish the effect of retrieval on future states. For systems jointly rewriting traces and controller state, this projection may not be applicable without a different intervention.

The separate closed-loop experiment addresses the feedback path that projection excludes. Each arm follows its own retrieval/update history across 1,280 synthetic streams, 11 arms, two readers, and 337,920 answers. Hybrid-minus-dense accuracy under the full controller is −14.86/−14.80 percentage points; all 12 frozen tests exclude a +2-point SDM benefit (Section 3.1; Table S10).

Importantly, this closed-loop result does not require a shared controller trajectory. It shows that retrieval-induced feedback does not reverse the substrate ordering on the tested generator, rather than establishing the same ordering for all systems with jointly evolving representations.

The two comparisons therefore serve complementary purposes: projection isolates the direct readout effect, while closed-loop evaluation tests whether the observed ordering persists when retrieval changes subsequent updates. The practical recommendation supported by both is to compare functioning, compact storage alternatives under the intended workload and resource constraints, rather than generalizing from a removal ablation or one storage representation.

W. 2: “Beyond the full-versus-one-step comparison, could linking and filtering be isolated and evaluated together with lifecycle costs under corresponding execution conditions? If this is outside the present scope, which practical recommendations should remain conditional?”

Response: We agree that the original full-versus-dense comparison attributes a gain to the complete retrieval bundle, while full-versus-one-step isolates only additional propagation. Neither identifies the separate contributions of linking and filtering. To investigate where the graph advantage arises, we added nested routing-policy comparisons and measured their answer quality and read costs in the same executions.

  • New routing-policy attribution. The new experiment produces 4,000 fresh answers across four policies, sharing 1,000 HotpotQA questions, 9,811 passages, the encoder, Qwen3.5-397B-A17B-FP8 reader, prompt, and top-five/6,000-evidence-token ceiling. Graph arms reuse the same constructed graph.
PolicyF1 (%)Model tokens per queryRead time per query (s)
Dense61.65785.330.713
Passage-seeded PPR60.56829.220.874
Raw-link PPR61.91805.550.892
Filtered-full67.803,856.836.786

The three planned adjacent F1 contrasts are:

Dense → Passage-seeded PPR: −1.09 [−2.64, 0.38] pp.

Passage-seeded PPR → Raw-link PPR: +1.35 [−1.74, 4.39] pp.

Raw-link PPR → Filtered-full: +5.89 [3.48, 8.35] pp.

These are multiplicity-adjusted 98.3333% intervals from 20,000 paired bootstrap draws over 988 shared-support components.

The clearest supported increment is therefore the implemented filter-and-fallback policy, which improves F1 by 5.89 points relative to Raw-link PPR. The two preceding increments establish neither superiority nor equivalence. We do not interpret their inconclusive intervals as evidence that linking or propagation makes no contribution.

This narrows the attribution substantially relative to full graph versus dense. Nevertheless, these are nested policy comparisons, not a complete linking-by-filtering factorial. In particular, the supported increment includes both filtering and its fallback behavior; it cannot be attributed to filtering alone. All 141 valid-empty-filter dense fallbacks remain included in the evaluation (Appendix S.2; Tables S.2–S.3).

  • The corresponding quality–cost tradeoff. Because answer quality and read costs are measured in the same executions, we can characterize the cost of the supported increment directly.

Relative to Raw-link PPR, Filtered-full improves F1 by 5.89 points while increasing model usage by approximately 3,051 tokens/query and read time by 5.894 seconds/query. Relative to Dense, Filtered-full uses 4.91× the model tokens and 9.52× the measured read time.

Token accounting includes prompts, completions, and filtering. Read time includes retrieval and reader phases, excluding initial construction and setup. Although the policies share the same evidence ceiling, their realized resource consumption is not equal.

This distinction matters for deployment: the supported quality gain is attributable to an implemented routing-policy increment, but obtaining it incurs substantial additional inference cost. It should therefore be evaluated against the application's quality requirements and resource constraints, rather than treated as an unconditional graph-retrieval advantage.

Corresponding initial lifecycle quality. We also scored all 9,000 saved initial-corpus answers from the three-arm, three-repetition lifecycle study, allowing answer quality to be paired with the build/read costs measured in those executions.

PolicyMean F1 (%)Median build plus 1,000 reads (s)
Dense60.901,904.23
Full graph67.9713,043.86

The full-minus-dense F1 difference is +7.07 points [5.14, 9.07], concentrated on bridge questions (+9.37 [7.20, 11.66]). These are post-hoc descriptive 95% intervals (Appendix Q).

This closes part of the quality–cost gap in the original lifecycle study: the measured initial construction and read costs can now be interpreted alongside answer quality from their corresponding executions. The lifecycle full-minus-one-step contrast remains uncertain (+0.73 [−0.46, 1.94] pp), and unlike the frozen prospective test, is not a pure propagation intervention.

We agree, however, that this is not a complete quality-adjusted lifecycle comparison. The graph and dense maintenance policies remain different (full rebuild versus native updates), and the 450 post-maintenance probes do not have updated gold answers. The reported 4.02-hour graph and 10.31-second dense maintenance costs characterize those specific policies; they do not establish an intrinsic maintenance disadvantage or a quality-adjusted deployment advantage for either architecture.

Accordingly, the practical recommendation is conditional. For bridge-heavy workloads, the implemented filter-and-fallback policy provides a supported answer-quality gain, but its additional read cost must be justified. The isolated value of linking, further propagation, and alternative maintenance policies remains unresolved. Our results identify an actionable quality–cost tradeoff, not a universally optimal routing architecture.

Q1. “Does the controlled diagnosis transfer to answer-level performance?”

We agree that the original RealMem result (+2.33 pp, 95% CI [−2.13, +7.07]) does not establish downstream superiority. To further examine transfer, we added a decay-by-routing factorial with 2,000 answers across 500 queries and ten simulated personas. The combined intervention yields −1.00 pp [−3.53, +1.20], with an interaction of +2.07 pp [−0.13, +5.27] (adjusted 97.5% intervals), while consuming 1.615× the baseline read tokens (Appendix S.3–S.4).

Neither benefit nor synergy is established. This additional experiment does not directly replicate the original storage intervention, and broader answer-level transfer remains unresolved. However, this uncertainty concerns downstream generalization, not the validity of the controlled effects under their stated conditions. Our contribution is to distinguish these levels of evidence rather than infer system-level superiority from component-level results.

Q.2. “Which practical recommendations are supported by the answer-level evidence?”

We further tested whether the temporal retrieval gains translate into final answers. On LongMemEval, re-clocking previously improved current-evidence update wins from 44/78 to 60/78. In the new 500-question answer-level evaluation, decay changes top-five retrieval on 384 questions, but correctness is 44.40% versus 45.20% without decay (−0.80 pp, 95% CI [−3.33, +1.62]; Appendix U). Historical comparisons sometimes require older evidence that decay demotes.

This demonstrates why mechanism activation and improved retrieval do not necessarily imply better final answers. It does not establish that decay is generally ineffective; the complete-history evaluation is also not an original-question-time deployment.

Accordingly, our recommendations remain tied to the measured evidence: compare compact storage alternatives under the intended budget; consider the filter-and-fallback routing gain alongside its additional cost; and validate temporal policies at the answer endpoint before deployment. We do not claim universal superiority or quality-adjusted maintenance benefits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.