- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that online hierarchical memory systems for LLM agents decide where each new record goes but never revisit that structure afterward. It claims this lets structural problems build up: a "magnet" node that pulls in most records through preferential attachment, near-duplicate sibling branches (fragmentation), and flat trees. Recall benchmarks such as LoCoMo and LongMemEval don't detect these problems. The proposed system, Gardener, has four parts:
- Anti-preferential placement. Candidate nodes are scored by code using per-keyword breadth voting divided by √(|v|+1). The LLM may only choose among these candidates.
- An online curation loop. It relocates, clusters, merges, splits, renames and cross-links nodes on nightly and reactive triggers. The LLM proposes each change, and deterministic code validates and applies it. Original records are never changed.
- Three-way ingestion triage. Each record is either placed, sent back to the user as a question ("ask-back"), or held as unclassified.
- Non-destructive "supersede" updates. Stale records are marked invalid, in the style of Zep/Graphiti bi-temporal invalidation, rather than rewritten or deleted.
The paper also proposes a set of structural-health metrics: magnet occupancy M, a title-Jaccard fragmentation index F, depth distribution, and unclassified rate. It pairs them with an oracle-vs-e2e "writer attribution" protocol taken from RoboMME.
Validation. The main tests use a 31-message multi-topic corpus derived from real usage. The corpus was originally in Korean, was translated to English, and is withheld. There are also 8 self-authored fixtures (51 questions). Baselines are MemTree (a reimplementation), Mem0 and A-MEM.
Results. Gardener's e2e recall (84.3%) is on par with the baselines (80–82%) and identical to the authors' own flat, no-tree control. Gardener cites stale values 0% of the time, against about 50% for the baselines. Its magnet occupancy is 31% at intake and 22% after one curation pass, against 42% for MemTree. The headline 64% collapse comes from the original backbone (Gemini 2.5 Flash), which was retired during the study. On the current backbone, turning off all guards gives only 33%. Out-of-domain probes are also reported: full LoCoMo with token-F1 0.331, and the LongMemEval knowledge-update and abstention subsets with 52/102.
Critical strengths
- S1. A real and under-measured problem, with a reasonable deterministic yardstick. Recall-only benchmarks can't see structural degradation, because an answer anywhere in memory still counts. The MemTree sweep (Fig. 4) gives concrete support: depth-adaptive thresholds alone trade one problem for another, either a magnet (θ₀=0.3 → M=55%) or deep fragmented chains (λ=0.1 → 24 branches, depth 8). The structural metrics are computed without an LLM, so they can be reproduced across systems. This framing could be useful for the memory-systems community even if the specific system turns out not to be.
S2. Non-destructive, code-applied supersede updates clearly remove stale-value citations. This is the paper's strongest empirical result: 0/18 stale citations in every run, against 50–56% for MemTree, Mem0 and A-MEM, with old values still answerable through a history query. To their credit, the authors attribute this to the supersede guard rather than to the tree, since the flat control also gets 0%. The general design rule ("the LLM proposes, code executes") is sound and practical. Some caveats apply to the size of this gain:
- MemTree has no update mechanism at all, so part of the gap is built into the comparison.
- Gardener declines to answer 15.7% of the time, while the baselines never do, and a non-answer can never cite a stale value.
- The probe has 18 author-written questions in which the updates are stated explicitly.
- On the real LongMemEval knowledge-update subset (38/72), stale-value citations are not reported.
Critical weaknesses
W1. On the primary backbone, the guards are not shown to bound structure, and the paper's own numbers contradict its gate claims.
- The intake numbers fail the gates. The gates are M ≤ 0.30 and unclassified ≤ 0.10 (§3.3). Intake reaches M = 31% (27–55% over 3 runs) and unfiled 33%, failing both. Yet three places claim otherwise:
- §6.2 says intake "stays within the health gate (magnet 31% at intake…)";
- §6.5 says all backbones "meet the structural-health gates";
- App. E lists "magnet 31%, unfiled 33%" and then says generation shifts surface "not as gate violations."
Only after curation (22%, 4%) are both gates met. The abstract should state that it refers to the post-curation state.
- Removing the guards makes no measurable difference. With all six guards off, M is 33%, against 31% with them on (Fig. 2). Removing anti-preferential scoring, contribution (1), leaves M unchanged on both backbones (31→31 and 20→20). Single ablations range from 25% to 42% at n=1, which is within run-to-run spread.
- The motivating collapse may not exist on current models. The 64% collapse appears only on a retired backbone, and only with all guards removed together. No bound is ever stated or measured: √-normalization weakens the advantage of large nodes but does not cap occupancy. The gates were also calibrated on the authors' own system, which makes passing them partly circular.
The abstract presents the backbone change as support for deterministic guards. In fact it weakens the intake-side contributions, (1) anti-preferential placement and (3) triage: neither changes intake M measurably on the current backbone (removing scoring gives 31→31; removing ask-back gives 25 vs 31). Only contribution (2), curation, has support, and only on one 31-record corpus (M 31% → 22%, unfiled 33% → 4%).
W2. Nothing shows that structural health matters for any task, and on the primary suite the tree adds nothing measurable over a flat store. The paper says poor structure hurts "branch-level context injection, summary coherence, and navigability," but none of these is measured. On the primary suite, Gardener e2e and the unstructured flat control are identical on every reported metric: recall 84.3%, stale citations 0%, avoidance 15.7%. The freshness benefit comes from supersede, which is independent of the hierarchy. So the experiments don't show that lower M or F leads to better agent behavior, and the structural metrics stay proxies with no demonstrated validity.
Two further problems weaken the metrics themselves:
- F = 0 for every tree system in Table 2, so the fragmentation index doesn't distinguish between systems.
- M can be trivially improved, for example by deep chains (MemTree λ=0.1 reaches 19%, below Gardener's intake).
A convincing version would vary structural health under controlled conditions and show a downstream effect, such as summary faithfulness, navigation or tool-use success, or retrieval cost.
Questions for authors
Clarifications. These can be answered from existing results, and the answers bear directly on W1 and S2:
- Is the health gate meant to apply only after curation? If so, the claims in §6.2, §6.5 and App. E should say so, and the intake-side guards would then have no measured effect. If not, how is intake M = 31% (up to 55%) with 33% unfiled "within the gate"? (This affects W1.)
- Is the curated 22% / 4% a median over independent runs or a single manual pass (Fig. 3)? Curation is the only component with measured support, so its variance matters. (If this is a single run, W1 and the rating are reinforced.)
- On the 18 knowledge-update questions, how many did Gardener decline to answer in each run, and what are its stale-value and accuracy rates on the questions it did answer? (If many were declined, the strength in S2 shrinks.)
- The Ethics Statement says no real-user data was used, yet the corpus is "derived from real usage." Which is correct? (This affects the ethics assessment.)
Additional evidence that could change my rating:
- With at least 5 runs per ablation cell and intervals, does any intake guard, in particular anti-preferential scoring, change M distinguishably from noise on the current backbone? Can the "bound enforced in code" be stated formally? (A positive answer would address W1.)
- Is there any downstream measure, such as summary faithfulness, navigation success, or token or latency cost, on which the tree beats the flat store? (A positive answer would address W2.)
5 comments
Reply to DuiZ (2 of 2): curation is one pass, now measured over five stores; E2: the tree beats flat on LongMemEval, and structure sets injection recall; stale rates; ethics resolved
Note: See our corrections of 15 Nov below.
(Part 2 of 2; terms and conditions are in part 1.)
W2, E2. Two downstream measures now separate the tree from the flat store. On LongMemEval, with the same writer and reader, the tree store answers 63 of 72 knowledge-update questions and the flat store 41, and the stale rates are 6.9% versus 28.3% (Q3 below). Under branch-level injection the session tree delivers the evidence. The submitted parity arose because the reader never used the tree: it searches all records, and branch injection was off. When context is injected by branch (each node contributes its roll-up, a digest capped at 20 lines), the structure decides which records reach the model. We stored the same 586 records in four shapes: a session tree (one node per session), one magnet node (every record in one node), a deep chain (each session nested under the previous one) and a flat store (no nodes). For each question we pinned (injected) the branch that holds the evidence turn:
| variant | evidence turn inside the injected block (193 questions, deterministic) |
|---|---|
| session tree | 67.9% |
| magnet | 0.5% |
| deep chain | 1.0% |
| flat, no pin | n/a |
Correction (13 Nov). This table first included pinned-reader F1 and fallback counts. They are withdrawn: in those runs, between 24 and 33 of the 56 reader replies per run were error replies returned while the model API was rate-limited, and our harness graded them as answers. The deterministic column above is unaffected. These runs are being repeated with retries, and corrected values will follow here.
The deep chain has a low magnet share yet almost never delivers the evidence turn, so magnet share must be read together with depth (W2-c). On W2-b, the raw fragment-pair count (title overlap) does discriminate on the public stores, 5 [2–8] at intake and 1 after curation, so we keep it; on the four shapes above it is 0 or 1 by construction. We add a content-level index: sibling roll-ups embedded, mean-centered per store and compared by cosine, with the threshold calibrated on split nodes (each node's roll-up cut in two gives a true same-topic pair). At the strict threshold, near-duplicate sibling pairs are 0 [0–3] at intake and 0 [0–2] after curation; at the lenient one, 11 [8–19] and 6 [2–7]. Where title discipline is off, one store counts 138 title pairs but 2 by content, so the revision reports both indices (Appendix D).
Q3, S2. On the 18 supersede probes, Gardener declined 1 of 18 in each of three runs, answered the other 17 correctly and cited no stale value. Flat declined 2, 2 and 0, answered the rest correctly and cited none. MemTree, Mem0 and A-MEM declined none, answered 5 to 9 of 18 correctly and cited a stale value in 8 to 11. MemTree has no update step, so its share of that gap is built in; Mem0 and A-MEM do update records and still cited 8 to 11. The declines in the submitted LongMemEval run were not abstentions but missing records: the writer's intent classifier, seeing its own earlier replies in the channel transcript, stopped recording, and every declined question had at most 5 records stored from about 47 sessions. The harness now delivers the haystack one user turn per message, without that transcript; MemTree, Mem0 and A-MEM ingest each session whole, their default usage. On LongMemEval knowledge-update questions, where the update is implicit, we grade the stale-citation rate among answered questions under one grader. It is Gardener 6.9% (4 of 58), MemTree 5.1% (3 of 59), A-MEM 13.3% (8 of 60), Mem0 32.6% (14 of 43) and the flat control 28.3% (15 of 53). Without a tree the same writer stored half as many records (median 86 versus 175 per question) and invalidated a quarter as many (22 versus 79 in total).
Q4 (Ethics Statement). Both sentences are true in different senses, and the first was written too broadly. "No real-user data" meant no data from anyone other than the authors; "derived from real usage" means that the workload is not synthetic. The facts are these. The 8 fixtures (51 questions, including the 18 supersede probes) are self-authored and synthetic and concern no real person. The 31-message workload is a work journal that one author kept through an AI-agent personal secretary during the study; no third-party users, participants or externally collected records are involved, so there was no human-subjects data collection. It contains no names, contact details or other identifiers of any third party in either language; organizations appear as placeholders and people only by role. The Ethics Statement now says exactly this, and the workload is not released. Since every current-backbone claim now rests on public corpora, this workload supports no claim beyond the historical observation on the retired backbone.
Corrections (15 Nov) to results in this thread, 1 of 2: pinned-reader F1 re-measured; no structure answers better than the flat store
Corrections (15 Nov). Four results in our replies to you change: the pinned-reader F1 withdrawn on 13 Nov, the LongMemEval counts and stale rates under W2 and Q3, the threshold count under Q1, and the replication under E1. This comment gives the first; our next comment in this thread gives the others. Terms: a pin injects one branch of the store into the reader's context for a question; injection recall is the share of questions whose evidence turn is inside the injected block. The same 586 records are stored in four shapes: a session tree (one node per session), a magnet node (every record in one node), a deep chain (each session nested under the previous one), and a flat store (no nodes).
- Pinned-reader token F1, re-measured. With the evidence branch pinned, no structure answers better than the flat store without pins. Median token F1 is 0.38 to 0.42 across the five pinned cells and 0.42 to 0.46 across the five cells without pins (flat store: 0.449). For the session tree and the deep chain, every pinned run lies below every run without pins. We withdraw the statement that pinned-reader F1 follows injection recall. The deterministic injection recall (67.9% under a session tree, 0.5% under a magnet node, 1.0% under a deep chain) is unchanged. The titles of our second replies to Reviewers DuiZ, rCfc, and 9kJE stated an advantage under injection, and we have edited them to remove it. On 13 Nov we edited four earlier comments in place to withdraw the F1 values first posted (0.361 for the session tree, 0.300 for the magnet node, and 0.265 for the deep chain, versus 0.312 for the flat store). In those runs, error replies returned while the model API was rate-limited had been graded as answers. From our second reply to you we also removed the fallback counts, the single-hop F1 values, and the retrieval calls per question, which came from the same runs. The values below replace them. We reran every cell with reader retries: three runs for each of the four shapes, and one run on each of the five stores that Gardener wrote and on their copies after curation.
| cell | runs | token F1, median [min–max] | questions with a fallback to global search, median [min–max] | retrieval calls per question | unanswered questions, min–max |
|---|---|---|---|---|---|
| session tree, pinned | 3 | 0.397 [0.379–0.403] | 9 [9–22] | 0.95 | 6–8 |
| magnet node, pinned | 3 | 0.407 [0.357–0.448] | 3 [3–27] | 3.07 | 8–12 |
| deep chain, pinned | 3 | 0.383 [0.375–0.386] | 30 [25–32] | 1.20 | 15–19 |
| Gardener stores, pinned | 5 | 0.422 [0.367–0.448] | 3 [1–5] | 0.41 | 3–8 |
| curated Gardener stores, pinned | 5 | 0.377 [0.354–0.438] | 5 [2–11] | 0.68 | 6–8 |
| session tree, no pin | 3 | 0.456 [0.453–0.459] | 0 | 1.34 | 8–11 |
| magnet node, no pin | 3 | 0.425 [0.399–0.455] | 0 | 1.77 | 9–12 |
| deep chain, no pin | 3 | 0.458 [0.427–0.480] | 0 | 1.73 | 7–8 |
| Gardener stores, no pin | 5 | 0.422 [0.370–0.462] | 0 | 0.77 | 3–8 |
| flat store, no pin | 3 | 0.449 [0.408–0.494] | 0 | 1.21 | 4–8 |
A fallback to global search is a query that found nothing inside the pinned branch and was repeated over the whole store; the column counts the questions with at least one. Retrieval calls per question counts the reader's search calls over all attempts at a question (up to eight). Single-hop F1, which we had posted as higher for the pinned session tree, is 0.49 for both the pinned session tree and the flat store.
After retries, no remaining failure comes from a rate-limited call. A reply that stays empty after eight attempts, because the reader stopped at its step limit without writing an answer, is scored as wrong; this happens to 3 to 19 of 56 questions per run.
The revised PDF, which we will upload later today, reports these values in Section 6.6 and in Table 7 of Appendix H.
Reply to DuiZ (1 of 2): corrections to the gate claims; E1 answered at n = 5, the hierarchy nudge, not scoring, prevents collapse, and the code bound is a score bound
Note: See our corrections of 15 Nov below.
We agree with W1 on the gate claims and on the single-run evidence. On the collapse itself, the public rerun below shows that it exists on the current backbone: with one guard off, a 586-turn conversation collapses to 7 nodes in five of five runs. We set the private corpus aside and rebuilt the evidence on public benchmarks during the discussion period, as E1 and E2 ask. On LoCoMo conversation 2 (586 turns) we ran seven intake cells at five runs each with ranges and an exact permutation test, replicated the decisive cell on two further conversations, curated five stores over three passes, and measured branch-level injection on four store shapes. On the LongMemEval knowledge-update subset (72 questions) we ran Gardener, a flat control and a single-call reader over the Gardener store alongside MemTree, Mem0 and A-MEM, all under one grader, with stale citations post-graded among answered questions. Every number below comes from these runs. Terms: magnet share is the fraction of records in the largest node; unclassified records are those left at the root; fragment pairs are sibling nodes with near-duplicate titles; an ask-back is a placement question put to the user. The hierarchy nudge asks the writer to open a sub-branch once a branch holds more than 8 direct records, and title discipline forbids sentence-like branch titles.
Q1, W1-a, W1-d. Five passages (abstract, Sections 6.2 and 6.5, conclusion, Appendix E) are wrong: they say that guarded intake meets the health gate, but only the post-curation state does. At intake, magnet share is 31% and unclassified share 33%; after curation they are 22% and 4%. We also withdraw the claim that the guards are model-independent, since the writer's recording behavior differs sharply across backbones. The thresholds are diagnostics, not controls: they were fixed in July 2026 before the evaluation, no code reads them, and the stores built from 10 of 10 LoCoMo dialogues and 100 of 102 LongMemEval questions meet the same thresholds.
W1-b/c/e, E1. On the evidence below, the guard claim narrows to one mechanism, the hierarchy nudge. First, the "bound enforced in code" is a bound on a candidate's score, not on a node's share of records. The keyword vote total raw(v) of candidate node v is at most 42, so score(v) is at most 42/sqrt(|v|+1) plus a recency term, where |v| is the node's record count. We reword the text to "volume-blind, size-penalized scoring". Second, sibling merge and bloat split (two of the six guards) run only during curation, so ablating them during ingestion could not show anything. On public data, two cells collapse the tree, one spreads it, and the other two match the baseline:
p = 0.048 is the smallest value an exact permutation test can give with five runs per cell. Turning off the hierarchy nudge alone reproduces the collapse. Three further runs each on conversations 3 and 6 replicate it under a rule fixed before the runs: magnet share 0.48 versus 0.044 on conversation 3 and 0.96 versus 0.034 on conversation 6, with every nudge-off run above every baseline run. Title discipline off spreads the tree instead (99 nodes, lower magnet share). The nudge is the one guard that reacts to branch size. We attribute the effect to this size feedback, not to the scoring term, and restrict the claim accordingly. The ask-back guard, contribution (3), leaves magnet share unchanged (p = 1.0) and acts on the unclassified share instead (Q2).
Q2. The curated 22% and 4% are one manual pass, not a median; Table 2 and Figure 3 now say so, and the node-count entry (13 to 11) pairs the three-run intake median with that single pass. On public stores curation does merge and split, which the submitted run never did (it only reassigned 7 records). On five public stores with three passes each, pass one reassigns a median of 10 records [4–16], merges 15 nodes [12–23], splits 7 nodes [6–8], and creates 2 new branches. Fragment pairs fall from 5 to 1, unclassified share from 2.1% to 0.2%, and node count from 83 to 69. Magnet share rises from 0.040 to 0.059 (0.088 after pass three) because merges enlarge the largest node; it stays under the 0.30 gate. Splitting is capped at two nodes per pass, so three passes do not converge. The curation claim now rests on the fragment-pair reduction. The drop in unclassified share comes mostly from resolved ask-backs, not from curation: turning ask-backs off alone gives 0.2% (p = 0.048).
(Continued in part 2.)