- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the memory medium deployed agents actually use — a directory tree of markdown files the agent itself writes and reorganizes — and tests two assumptions the practice rests on: that an agent can keep such a memory organized as it grows, and that organizing pays. Section 2 formalizes the setting as three roles (management, search, execution) acting on one memory filesystem through an interchangeable harness, with declarative content and skills in the same store, and states a five-property "taxonomy contract" as its working definition of good organization. Sections 3–4 instantiate this on long-conversation QA with attribution and on embodied task success, varying memory variant, stream scale, harness, and the backbones of the two memory agents, and report answer quality, cost, and memory structure together.
Critical strengths
Strength 1: The variant ladder in Sec. 3 is the right design for the question. Foldered sessions relocates files without editing their bytes while Reorganized memory rewrites them, and the preserve/condense pair differs by one instruction. This separates layout from content fidelity, which is what makes the dissociation in Table 1 — organization reliably buying search economy while no organization leads correctness — a result rather than a benchmark accident, and it is what exposes free reorganization as silently lossy (Sec. 4.1.1).
Strength 2: Treating the harness as a first-class axis (Sec. 4.1.3) is a new and practically useful control. Showing that the tool layer reshapes memory structure as strongly as a backbone step, and that in the embodied setting replacing it moves success by as much as a backbone step does (Sec. 4.2.3, Table 2), turns the filesystem default into something a designer can act on rather than only observe.
Critical weaknesses
Weakness 1: The headline nulls rest on single build draws, and the cells carrying the sharpest claims are small. Both PersonaMem tiers — the basis of the curation collapse and of the backbone grid in Table 3 — rest on a few dozen questions, against a judge noise floor the authors measure in Table 1's footnote, and Sec. 4.1.3 reports two builds differing only in a corrected tool description landing an order of magnitude apart in file count. So the quality comparisons cannot separate "organization does not affect answers" from "this design cannot see the effect", though the former carries the abstract. That two of the four research questions are answered only in the appendix compounds the difficulty of checking this.
Weakness 2: The taxonomy contract is the paper's own normative standard, yet the erosion result in Figure 3 is drift in a lexical descriptor whose link to utility the authors themselves leave open and which, as they note, leans with granularity. The memory that holds its adherence is also the one built by the strongest backbone and the one that stopped creating files, so the reading cannot be separated from the backbone. The issue is not the disclosure but the weight: a sustainability claim reaches the abstract on a measure no experiment ties to retrieval outcomes.
Questions for authors
Question 1: On one benchmark, could you run a few repeated builds per harness and per management-agent tier and report the spread in both structure and correctness? If build-to-build spread covers the between-cell spread, the organization-independence claim should be stated as an upper bound on the effect; if it does not, the claim becomes considerably stronger.
Question 2: Foldered sessions supplies a layout-only addition control; its mirror image is missing. Could you degrade a well-organized memory with content held fixed — permuting files across folders, or flattening it — and measure search cost, correctness, and the contract metrics on the same questions? A move in cost or accuracy with no move in the metrics would indicate the measures miss what makes a layout useful; no move in either would make "organization does not pay here" a much firmer claim.
5 comments
Response to Reviewer f9ti (2/3)
Q2: Degrading organization while preserving the text
Thank you for suggesting this control. We tested both folder layout and the assignment of sections to files. Scattering sections across files increased search cost in all three many-file memories, even though the text was preserved. Folder changes alone had no consistent effect on accuracy or cost. These controls separate the contribution of folder placement from that of assigning content to files.
Folder layout. We flattened or randomly reassigned files in three PersonaMem 128k memories, preserving their contents except for updating path references. We used the same 42 questions and search agent, and repeated searches of the original layouts.
| Memory | Layout | Correct of 42 | Cost (cents/query) | B4 |
|---|---|---|---|---|
| GPT-5.4-nano, 122 files | Original | 29, 29 | 5.4, 5.1 | 0.25 |
| Flattened | 32 | 4.6 | 0.21 | |
| Random | 29 | 4.1 | 0.19 | |
| GPT-5.4, 105 files | Original | 30, 31 | 4.7, 5.0 | 0.18 |
| Flattened | 29 | 3.9 | 0.02 | |
| Random | 29 | 4.5 | 0.05 | |
| Shell, 147 files | Original | 28, 31, 25 | 3.7, 4.0, 3.5 | 0.23 |
| Flattened | 27 | 3.7 | 0.09 | |
| Random, first layout | 22, 23 | 4.5, 4.9 | 0.12 | |
| Random, second layout | 29 | 3.2 | 0.07 |
Both tables in this comment price cost without caching because cache shares differed across runs. B4 measures whether tree distance mirrors lexical relatedness. Here it uses all content-unit pairs, whereas Table A6 uses a sample.
B4 fell in every changed layout, while the other three contract metrics (B1 to B3) barely moved. Accuracy and cost did not change consistently. One random shell layout scored 22 and 23, just below the original's 25 to 31, but a second scored 29, within that range. The folder test is the mirror image of Foldered sessions (Table 1) and agrees with it: neither adding nor removing folders produced a consistent change in accuracy or cost.
Sections within files. We then randomly reassigned leaf sections across files in these three memories and the two-file curated memory from Table 1. All text, file/folder names, descriptions, and path references were retained. Each memory had two new searches of the original and two searches of each of two shuffles.
| Memory | Correct of 42: original | Correct of 42: shuffled | Cost (cents/query): original | Cost (cents/query): shuffled | B2 | B4 |
|---|---|---|---|---|---|---|
| GPT-5.4-mini, 2 files | 28, 31 | 30, 31, 29, 30 | 4.0, 3.1 | 4.0, 3.9, 4.3, 3.7 | 2.2 to 1.0 | 0.16 to -0.03 |
| GPT-5.4-nano, 122 files | 30, 30 | 30, 27, 27, 31 | 5.4, 4.5 | 9.1, 8.9, 7.8, 8.7 | 4.5 to 1.0 | 0.25 to 0.01 |
| GPT-5.4, 105 files | 32, 30 | 29, 28, 29, 27 | 5.3, 5.5 | 8.8, 7.6, 10.1, 10.0 | 4.9 to 1.0 | 0.18 to -0.03 |
| Shell, 147 files | 26, 26 | 32, 27, 29, 29 | 4.2, 4.2 | 5.7, 6.5, 7.5, 6.9 | 5.3 to 1.0 | 0.23 to 0.04 |
B2 (sibling content cohesion) and B4 show original to mean-shuffled values, computed over all eligible sections and pairs rather than the sample used in Table A8, which gives B2 of 2.50, 4.95 and 4.50 for the first three originals, in this order. With sections placed at random, B2 is about 1 and B4 about 0.
Every shuffled search of the three many-file memories costs more than either original search; per memory, the average increase is roughly 60% to 75% without caching, or 40% to 55% as billed. In the two many-file memories searched with file tools, the agent opens one to two more files per question that it does not cite. The two-file memory shows a smaller, overlapping cost change. Correctness falls in two memories and rises in two, averaging 29.1/42 both before and after.
B2 and B4 detect the section shuffles, and the same intervention raises search cost in all three many-file memories. The two-file and folder controls show why these metric changes should be interpreted alongside measured cost and accuracy. B1 (label distinctness) rises slightly after section shuffling, while B3 (scope leakage) does not register it. B3 includes each section in its own group's centroid, which may contribute to this insensitivity. In your terms, the folder layout is close to your second case; when the sections of the many-file memories are shuffled, B2 and B4 move together with cost, and B3 is your first case. The revised paper will describe this result and qualify B3's ability to detect misplaced content.
The section test disrupts both topical grouping and the match between contents and file names, descriptions, or references, so it cannot separate their contributions. It shows that retrieval becomes more expensive when the sections of a many-file memory are scattered across its files, even with the text preserved. Both tests cover one conversation, 42 questions, and one search agent.
In the revision, we will present the folder and section controls together, with all four contract metrics alongside accuracy and cost.
Response to Reviewer f9ti (3/3)
W2: Interpreting the erosion result
We appreciate the distinction between structural drift and practical usefulness. The abstract's sentence on sustainability rests first on outcomes: a growing memory "stays useful and its structure holds" (task success against no memory, survival of early files; App. E.3). The erosion is its closing clause, and we agree that it should be read as a description of one lexical measure. We examined what drives the decline of B4, the lexical layout-content measure of Figure 3. In the five growing chains we examined, B4 falls for the whole memory while remaining largely stable for files present at task 40 and at task 140.
The analysis covers the five Table A28 chains with B4 of at least 0.19 at task 40 that add at least three files by task 140. Mean B4 for the whole memory falls from 0.32 to 0.22 between these checkpoints. For files present at both, it remains at 0.33 and 0.34, with a decline in only one chain (0.25 to 0.21). This is consistent with dilution as new files join a nearly flat layout, while existing files largely retain their lexical fit. The pattern appears under both management models represented in these five chains.
The Q2 experiments help interpret this result. B4 detects every layout perturbation, while search cost rises consistently only when the sections of a many-file memory are scattered across its files, and accuracy changes in different directions. The structural analysis describes how the memory changes, while task outcomes establish whether it remains useful. A lower B4 alone does not establish a loss of utility, which supports the weight you ask for. The paper introduces its contract metrics, B4 among them, as "descriptors of contract adherence, not quality scores" (App. A.2). These interventions test conversational memories, and the embodied chains in Figure 3 were not perturbed.
The memory that holds its B4, the strongest management model's, is the one whose agent shifted early from creating files to editing them, which is, as the paper states, "an association these chains cannot separate from the backbone itself". We will retain this qualification.
In the revision, we will add the matched-file analysis to the growth discussion, and the figure's caption will identify B4 as a lexical layout-content measure. In the revised abstract, the clause will describe the decline as new files diluting a lexical measure of how closely the layout follows the content, while the files already written largely keep their fit. This will preserve the finding about growth while making clear what the lexical decline does and does not establish.
Ethics flag
The study uses publicly released benchmarks, recruits no new participants, and collects no new personal data, as stated in the Ethics statement. We would be glad to address any specific concern behind the flag.
We hope this addresses your concerns and are happy to clarify further.
Response to Reviewer f9ti (1/3)
We thank the reviewer for the careful review, and for valuing the variant ladder that separates layout from content fidelity and the treatment of the harness as an axis of its own.
W1 & Q1: Repeated builds and larger evaluation cells
Thank you for suggesting repeated builds to separate variation within a setting from differences between settings. We ran this experiment on PersonaMem 128k: two additional builds per management-model/tool setting, with each new build searched twice.
The repeated builds preserve the main structural contrasts between management models. More elaborate structures do not consistently yield more correct answers in these runs.
Files and sections list the original build followed by the two new builds. Correctness lists two searches of each new build. The semicolon separates the builds. Searches run one question at a time.
With the same file tools, GPT-5.4-mini consistently produces fewer files and sections than the other management models. Ranked search also produces fewer sections and a shallower hierarchy than the shell in every build. The shell's file count ranges from 4 to 147. We will describe these recurring structural tendencies together with their variation across builds.
Two searches of the same memory differ by 1.8 answers on average. The five setting means span 2.5 answers (6 percentage points). The management-model and tool-set accuracy differences are therefore comparable to repeated-search variation. We will state them as an upper bound on the effect, as you suggest, rather than infer zero effect. Search-agent capability shows a clearer difference in Table 3: across its three memories, the strongest search agent wins on 15 questions and loses on 4 against the weakest (question-paired sign test, p = 0.02).
We also evaluated the Table 1 filesystem variants on eleven additional PersonaMem conversations, covering 166 distinct questions:
Each cell uses one build and one search with the paper's models, prompts, and tools, run one question at a time. The 32k scores use distinct questions, accounting for duplicates among the 111 question rows.
The dissociation you highlight as a strength remains on this larger sample. Reorganized memories cost 43% to 55% less to search than verbatim files (30% to 40% without caching), but rewritten memories do not improve accuracy in either tier. The 32k deficits are concentrated in two conversations. The larger 128k deficits occur in both conversations.
The additional conversations put the original 32k result in perspective. That cell contains 19 distinct questions in 32 rows, and its unusually large curation deficit is not consistent across repeat runs. The additional conversations show a narrower gap. We will use the expanded evaluation for the main 32k comparison and report the original result and the reruns as well.
The nine 32k personas were selected by a rule fixed in advance, outside the paper's test and validation splits because no further 32k conversation was available in the test split. The 128k conversations are from the test split. Twenty questions are shared with conversations we had used before (the paper's own, and one from our June pilots), and excluding them leaves the pattern unchanged. Five reorganized memories were rebuilt once after interruption or failure of preset validity checks. For one memory, neither attempt met every check, so we retained the first. No build was selected by answer quality.
In the revision, the abstract and the main text will distinguish reproducible structural differences, accuracy differences between management models and between tool sets that are comparable to repeated-search variation, and the search-cost savings supported by the expanded sample. The main conversational tables will include uncertainty intervals, and we will move the main growth and cross-family findings into the main text so all four research questions can be assessed there.