acceptodds
Under review as a conference paper at ICLR 2027

Filesystem-Based Memory for LLM Agents: Organization, Evolution, and Sustainability

Abstract

Deployed large language model (LLM) agents increasingly keep their long-term memory as a filesystem: a directory tree of markdown files that the agent itself reads, writes, and reorganizes through generic file tools. Yet research has largely passed over this medium: prior systems design bespoke memory representations and study retrieval over them, leaving the default's two working assumptions untested: that an agent can keep a growing memory organized as memories accumulate, conflict, and go stale, and that this organization pays. We present, to our knowledge, the first systematic exploration of filesystem-based memory for LLM agents. We formalize the setting as three roles around one memory filesystem: a management agent integrates and organizes incoming content, a search agent answers queries with cited sources, and an execution agent supplies task trajectories that are distilled into skills, unifying declarative memory and skills in the same filesystem. Across long-conversation benchmarks and embodied tasks, we vary the memory variant, the stream's scale, the harness, and the backbones of the management and search agents, tracking answer quality, cost, and the health of the memory as it grows. Organization reliably buys search economy, but not always answer quality: reorganized memories more than halve per-query search cost on the two PersonaMem tiers, where the memorized content is largest, while no memory organization leads answer quality everywhere. As a memory grows it stays useful and its structure holds, but its fit to the content erodes in the well-organized memories that keep adding files. Capability matters in the agent that reads on conversation and, as a step at the top tier, in the agent that writes in the embodied setting, and two other model families also show no universally best organization while changing its form and its benefits. The harness alone reshapes the memory as strongly as the model. The study turns the filesystem default from an assumption into a design space for agent memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, filesystem-based memory, memory organization, skill libraries

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f9tiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the memory medium deployed agents actually use — a directory tree of markdown files the agent itself writes and reorganizes — and tests two assumptions the practice rests on: that an agent can keep such a memory organized as it grows, and that organizing pays. Section 2 formalizes the setting as three roles (management, search, execution) acting on one memory filesystem through an interchangeable harness, with declarative content and skills in the same store, and states a five-property "taxonomy contract" as its working definition of good organization. Sections 3–4 instantiate this on long-conversation QA with attribution and on embodied task success, varying memory variant, stream scale, harness, and the backbones of the two memory agents, and report answer quality, cost, and memory structure together.

Critical strengths

Strength 1: The variant ladder in Sec. 3 is the right design for the question. Foldered sessions relocates files without editing their bytes while Reorganized memory rewrites them, and the preserve/condense pair differs by one instruction. This separates layout from content fidelity, which is what makes the dissociation in Table 1 — organization reliably buying search economy while no organization leads correctness — a result rather than a benchmark accident, and it is what exposes free reorganization as silently lossy (Sec. 4.1.1).

Strength 2: Treating the harness as a first-class axis (Sec. 4.1.3) is a new and practically useful control. Showing that the tool layer reshapes memory structure as strongly as a backbone step, and that in the embodied setting replacing it moves success by as much as a backbone step does (Sec. 4.2.3, Table 2), turns the filesystem default into something a designer can act on rather than only observe.

Critical weaknesses

Weakness 1: The headline nulls rest on single build draws, and the cells carrying the sharpest claims are small. Both PersonaMem tiers — the basis of the curation collapse and of the backbone grid in Table 3 — rest on a few dozen questions, against a judge noise floor the authors measure in Table 1's footnote, and Sec. 4.1.3 reports two builds differing only in a corrected tool description landing an order of magnitude apart in file count. So the quality comparisons cannot separate "organization does not affect answers" from "this design cannot see the effect", though the former carries the abstract. That two of the four research questions are answered only in the appendix compounds the difficulty of checking this.

Weakness 2: The taxonomy contract is the paper's own normative standard, yet the erosion result in Figure 3 is drift in a lexical descriptor whose link to utility the authors themselves leave open and which, as they note, leans with granularity. The memory that holds its adherence is also the one built by the strongest backbone and the one that stopped creating files, so the reading cannot be separated from the backbone. The issue is not the disclosure but the weight: a sustainability claim reaches the abstract on a measure no experiment ties to retrieval outcomes.

Questions for authors

Question 1: On one benchmark, could you run a few repeated builds per harness and per management-agent tier and report the spread in both structure and correctness? If build-to-build spread covers the between-cell spread, the organization-independence claim should be stated as an upper bound on the effect; if it does not, the claim becomes considerably stronger.

Question 2: Foldered sessions supplies a layout-only addition control; its mirror image is missing. Could you degrade a well-organized memory with content held fixed — permuting files across folders, or flattening it — and measure search cost, correctness, and the contract metrics on the same questions? A move in cost or accuracy with no move in the metrics would indicate the measures miss what makes a layout useful; no move in either would make "organization does not pay here" a much firmer claim.

5 comments
AuthorsOpenReview ↗

Response to Reviewer f9ti (1/3)

We thank the reviewer for the careful review, and for valuing the variant ladder that separates layout from content fidelity and the treatment of the harness as an axis of its own.

W1 & Q1: Repeated builds and larger evaluation cells

Thank you for suggesting repeated builds to separate variation within a setting from differences between settings. We ran this experiment on PersonaMem 128k: two additional builds per management-model/tool setting, with each new build searched twice.

The repeated builds preserve the main structural contrasts between management models. More elaborate structures do not consistently yield more correct answers in these runs.

Management model and toolsFilesSectionsCorrect of 42
GPT-5.4-nano, file tools122, 148, 111413, 519, 46531, 31; 33, 31
GPT-5.4-mini, file tools2, 38, 2210, 245, 21033, 31; 32, 30
GPT-5.4, file tools105, 100, 79561, 523, 46129, 31; 28, 31
GPT-5.4-mini, ranked-search file tools1, 3, 1147, 99, 19230, 30; 31, 30
GPT-5.4-mini, shell147, 43, 4421, 282, 24331, 26; 29, 30

Files and sections list the original build followed by the two new builds. Correctness lists two searches of each new build. The semicolon separates the builds. Searches run one question at a time.

With the same file tools, GPT-5.4-mini consistently produces fewer files and sections than the other management models. Ranked search also produces fewer sections and a shallower hierarchy than the shell in every build. The shell's file count ranges from 4 to 147. We will describe these recurring structural tendencies together with their variation across builds.

Two searches of the same memory differ by 1.8 answers on average. The five setting means span 2.5 answers (6 percentage points). The management-model and tool-set accuracy differences are therefore comparable to repeated-search variation. We will state them as an upper bound on the effect, as you suggest, rather than infer zero effect. Search-agent capability shows a clearer difference in Table 3: across its three memories, the strongest search agent wins on 15 questions and loses on 4 against the weakest (question-paired sign test, p = 0.02).

We also evaluated the Table 1 filesystem variants on eleven additional PersonaMem conversations, covering 166 distinct questions:

Correct (%) / search cost (cents/query, as billed)Verbatim filesFoldered verbatim filesReorganized (preserve; condense)Curated
Nine 32k conversations, 96 distinct questions68.2 / 2.772.4 / 3.064.1 / 1.5; 60.4 / 1.364.1 / 1.9
Two 128k conversations, 70 questions67.1 / 3.665.7 / 3.547.1 / 1.9; 48.6 / 1.650.0 / 2.8

Each cell uses one build and one search with the paper's models, prompts, and tools, run one question at a time. The 32k scores use distinct questions, accounting for duplicates among the 111 question rows.

The dissociation you highlight as a strength remains on this larger sample. Reorganized memories cost 43% to 55% less to search than verbatim files (30% to 40% without caching), but rewritten memories do not improve accuracy in either tier. The 32k deficits are concentrated in two conversations. The larger 128k deficits occur in both conversations.

The additional conversations put the original 32k result in perspective. That cell contains 19 distinct questions in 32 rows, and its unusually large curation deficit is not consistent across repeat runs. The additional conversations show a narrower gap. We will use the expanded evaluation for the main 32k comparison and report the original result and the reruns as well.

The nine 32k personas were selected by a rule fixed in advance, outside the paper's test and validation splits because no further 32k conversation was available in the test split. The 128k conversations are from the test split. Twenty questions are shared with conversations we had used before (the paper's own, and one from our June pilots), and excluding them leaves the pattern unchanged. Five reorganized memories were rebuilt once after interruption or failure of preset validity checks. For one memory, neither attempt met every check, so we retained the first. No build was selected by answer quality.

In the revision, the abstract and the main text will distinguish reproducible structural differences, accuracy differences between management models and between tool sets that are comparable to repeated-search variation, and the search-cost savings supported by the expanded sample. The main conversational tables will include uncertainty intervals, and we will move the main growth and cross-family findings into the main text so all four research questions can be assessed there.

AuthorsOpenReview ↗

Response to Reviewer f9ti (2/3)

Q2: Degrading organization while preserving the text

Thank you for suggesting this control. We tested both folder layout and the assignment of sections to files. Scattering sections across files increased search cost in all three many-file memories, even though the text was preserved. Folder changes alone had no consistent effect on accuracy or cost. These controls separate the contribution of folder placement from that of assigning content to files.

Folder layout. We flattened or randomly reassigned files in three PersonaMem 128k memories, preserving their contents except for updating path references. We used the same 42 questions and search agent, and repeated searches of the original layouts.

MemoryLayoutCorrect of 42Cost (cents/query)B4
GPT-5.4-nano, 122 filesOriginal29, 295.4, 5.10.25
Flattened324.60.21
Random294.10.19
GPT-5.4, 105 filesOriginal30, 314.7, 5.00.18
Flattened293.90.02
Random294.50.05
Shell, 147 filesOriginal28, 31, 253.7, 4.0, 3.50.23
Flattened273.70.09
Random, first layout22, 234.5, 4.90.12
Random, second layout293.20.07

Both tables in this comment price cost without caching because cache shares differed across runs. B4 measures whether tree distance mirrors lexical relatedness. Here it uses all content-unit pairs, whereas Table A6 uses a sample.

B4 fell in every changed layout, while the other three contract metrics (B1 to B3) barely moved. Accuracy and cost did not change consistently. One random shell layout scored 22 and 23, just below the original's 25 to 31, but a second scored 29, within that range. The folder test is the mirror image of Foldered sessions (Table 1) and agrees with it: neither adding nor removing folders produced a consistent change in accuracy or cost.

Sections within files. We then randomly reassigned leaf sections across files in these three memories and the two-file curated memory from Table 1. All text, file/folder names, descriptions, and path references were retained. Each memory had two new searches of the original and two searches of each of two shuffles.

MemoryCorrect of 42: originalCorrect of 42: shuffledCost (cents/query): originalCost (cents/query): shuffledB2B4
GPT-5.4-mini, 2 files28, 3130, 31, 29, 304.0, 3.14.0, 3.9, 4.3, 3.72.2 to 1.00.16 to -0.03
GPT-5.4-nano, 122 files30, 3030, 27, 27, 315.4, 4.59.1, 8.9, 7.8, 8.74.5 to 1.00.25 to 0.01
GPT-5.4, 105 files32, 3029, 28, 29, 275.3, 5.58.8, 7.6, 10.1, 10.04.9 to 1.00.18 to -0.03
Shell, 147 files26, 2632, 27, 29, 294.2, 4.25.7, 6.5, 7.5, 6.95.3 to 1.00.23 to 0.04

B2 (sibling content cohesion) and B4 show original to mean-shuffled values, computed over all eligible sections and pairs rather than the sample used in Table A8, which gives B2 of 2.50, 4.95 and 4.50 for the first three originals, in this order. With sections placed at random, B2 is about 1 and B4 about 0.

Every shuffled search of the three many-file memories costs more than either original search; per memory, the average increase is roughly 60% to 75% without caching, or 40% to 55% as billed. In the two many-file memories searched with file tools, the agent opens one to two more files per question that it does not cite. The two-file memory shows a smaller, overlapping cost change. Correctness falls in two memories and rises in two, averaging 29.1/42 both before and after.

B2 and B4 detect the section shuffles, and the same intervention raises search cost in all three many-file memories. The two-file and folder controls show why these metric changes should be interpreted alongside measured cost and accuracy. B1 (label distinctness) rises slightly after section shuffling, while B3 (scope leakage) does not register it. B3 includes each section in its own group's centroid, which may contribute to this insensitivity. In your terms, the folder layout is close to your second case; when the sections of the many-file memories are shuffled, B2 and B4 move together with cost, and B3 is your first case. The revised paper will describe this result and qualify B3's ability to detect misplaced content.

The section test disrupts both topical grouping and the match between contents and file names, descriptions, or references, so it cannot separate their contributions. It shows that retrieval becomes more expensive when the sections of a many-file memory are scattered across its files, even with the text preserved. Both tests cover one conversation, 42 questions, and one search agent.

In the revision, we will present the folder and section controls together, with all four contract metrics alongside accuracy and cost.

AuthorsOpenReview ↗

Response to Reviewer f9ti (3/3)

W2: Interpreting the erosion result

We appreciate the distinction between structural drift and practical usefulness. The abstract's sentence on sustainability rests first on outcomes: a growing memory "stays useful and its structure holds" (task success against no memory, survival of early files; App. E.3). The erosion is its closing clause, and we agree that it should be read as a description of one lexical measure. We examined what drives the decline of B4, the lexical layout-content measure of Figure 3. In the five growing chains we examined, B4 falls for the whole memory while remaining largely stable for files present at task 40 and at task 140.

The analysis covers the five Table A28 chains with B4 of at least 0.19 at task 40 that add at least three files by task 140. Mean B4 for the whole memory falls from 0.32 to 0.22 between these checkpoints. For files present at both, it remains at 0.33 and 0.34, with a decline in only one chain (0.25 to 0.21). This is consistent with dilution as new files join a nearly flat layout, while existing files largely retain their lexical fit. The pattern appears under both management models represented in these five chains.

The Q2 experiments help interpret this result. B4 detects every layout perturbation, while search cost rises consistently only when the sections of a many-file memory are scattered across its files, and accuracy changes in different directions. The structural analysis describes how the memory changes, while task outcomes establish whether it remains useful. A lower B4 alone does not establish a loss of utility, which supports the weight you ask for. The paper introduces its contract metrics, B4 among them, as "descriptors of contract adherence, not quality scores" (App. A.2). These interventions test conversational memories, and the embodied chains in Figure 3 were not perturbed.

The memory that holds its B4, the strongest management model's, is the one whose agent shifted early from creating files to editing them, which is, as the paper states, "an association these chains cannot separate from the backbone itself". We will retain this qualification.

In the revision, we will add the matched-file analysis to the growth discussion, and the figure's caption will identify B4 as a lexical layout-content measure. In the revised abstract, the clause will describe the decline as new files diluting a lexical measure of how closely the layout follows the content, while the files already written largely keep their fit. This will preserve the finding about growth while making clear what the lexical decline does and does not establish.

Ethics flag

The study uses publicly released benchmarks, recruits no new participants, and collects no new personal data, as stated in the Ethics statement. We would be glad to address any specific concern behind the flag.

We hope this addresses your concerns and are happy to clarify further.

Reviewer_NRVROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper abstracts interactions with the file-system-based memory into three distinct agent roles, which perform operations via a set of interchangeable tools (such as creating, rewriting, merging, or deleting files):

  • Management Agent: Responsible for writing, summarizing, restructuring, and cleaning up files, as well as maintaining the structure of the file tree.
  • Search/Retrieval Agent: Responsible for navigating the file tree and searching for required information.
  • Execution Agent: Responsible for utilizing the retrieved knowledge to complete specific downstream tasks.

This paper demonstrates through large-scale empirical research that file systems are an excellent transparent medium for managing Agent memory (which can reduce retrieval token consumption). However, maintaining a perfect folder structure long-term through upper-level Agent management is not only costly but also inevitably leads to an "entropy crisis." At the same time, the benefits of structured organization are highly dependent on the strength of the downstream consumption model.

Critical strengths
  1. By elevating filesystem organization to an explicit design space with operationalized metrics (e.g., tree-wide proximity, parent-child coverage), the authors provide a rigorous benchmark framework for agentic memory maintenance. It proved its strenth in a carefully designed evaluation setting.

  2. It explicitly points out the counterproductive effect that "curated knowledge", such as information that has been overly simplified or distilled, can have on powerful models (such as top-tier, high-capacity models). The paper avoids a common bias in the agent-memory literature. Specifically, demonstrating that frontier execution backbones achieve higher task success on raw, uncurated episodic trajectories than on curated, digested files is a vital insight.

  3. By stress-testing the memory repository over an extended 140-task horizon, the authors demonstrated a downgrade in the taxonomy, showing that unconstrained LLM management agents inevitably lead to structural entropy without active self-balancing.

Critical weaknesses
  1. The authors contrast append-only raw logging with curated file maintenance. This echoes the classic trade-off between write-optimized architectures and read-optimized systems, a common system-design dilemma. Under strict system and database standards, this paper neither considered nor could withstand more than 1000 consecutive interactions (far fewer than industry-level agents) at the design and experimental levels. Its testing and validation of the Episode Log (the raw interaction log that only increments and never decrements) clearly show a huge limitation. Why? At the scale of ~100 steps, the total number of tokens generated from the original log file (Markdown/JSON) is typically in the hundreds of thousands to a million. For top-tier large models (such as GPT/Claude/Gemini) that generally have a context window of 128K or even 1M+, the 140-step process falls into a sweet spot where "even if the retrieval agent is poorly made, the large model can still read a large amount of context by forcibly absorbing it."

  2. The paper emphasizes that "content-related entities should be close in distance within a tree", which is good in general. However, high-dimensional semantic spaces cannot be projected losslessly onto a single-rooted tree topology. For example: "An error log about an asynchronous web crawler in Python" should be placed in the /python/async/, /network/spider/, or /errors/ directory? Over decades of related work, the database community has shown that hierarchical databases lag far behind relational and graph models in expressing complex many-to-many semantics. Forcing an agent to achieve global proximity on a single physical tree is essentially fitting a high-dimensional topological manifold to a single-rooted tree, which mathematically causes the classification contract to collapse, unfortunately.

  3. Lack of rollback capability: When the Management Agent hallucinates, accidentally deletes or incorrectly rewrites a core file or breaks the existing directory structure, the system lacks atomic version control, like WAL (Write-Ahead Logging) in relational databases or file systems, to perform snapshot rollback.

  4. The Retrieval Agent performs multi-step tool calls (ls/cd) on the file tree. Since each tool call is accompanied by significant network latency, LLM generation latency, and context reread overhead, this top-down, layer-by-layer probing without statistical estimation raises concerns about overall system latency, given the contribution claim made by the authors.

Questions for authors
  1. Why did the paper arrive at the biased conclusion that "Episode Log is better than organized memory"? Was it solely because the experiment was truncated within a time window that was extremely favorable to Episode Log?

  2. Why did the authors restrict the system to a rigid directory structure rather than adopting standard OS filesystem primitives, such as link tools and tag-based indexing, which would natively resolve the multi-inheritance conflict?

  3. What is the average wall-clock latency and API invocation count per retrieval task under the organized filesystem compared to flat log inspection or simple BM25/embedding retrieval?

  4. The paper emphasizes that hierarchical organization cuts per-query retrieval token consumption. However, this comparison appears to be an incomplete accounting of overall system cost. To maintain taxonomy compliance, the Management Agent must repeatedly read multiple candidate files, synthesize summaries, and emit rewritten .md/.json/.yaml structures across lifecycle updates. What is the cumulative end-to-end token consumption (Management Write/Rewrite Tokens + Retrieval Tokens + Execution Tokens) across the entire 140-task horizon?

4 comments
AuthorsOpenReview ↗

Response to Reviewer NRVR (1/3)

We thank the reviewer for the careful review, and for valuing the measures of organization, the finding on raw episodes with strong execution agents, and the 140-task stress test.

W1 & Q1: Episode Log and the 140-task horizon

We appreciate the concern that a short horizon could favor raw logs. Our conclusion is that the value of curation depends on the agents involved. A curated variant has the highest observed success in 5 of 6 settings (three model families, two execution agents; Table A30). The change across agents is part of the finding: whether rewriting helps depends on what it preserves and what the later agent can use. The write/read tradeoff motivates this question, and the experiments examine how rewriting affects the information available for later tasks.

The log is accessed through selective retrieval (2.5 to 3.9 episode files returned per task; App. E.3), so a larger store does not mean the whole history is placed in context. The 140 tasks are full episodes, rather than individual interaction steps: 3,055 and 2,177 consecutive actions in the two Episode log chains of Table A15.

To test retrieval with a larger store, we added 930 training episodes generated without memory to the 70-episode log built from GPT-4.1-mini's episodes (Table A18). We evaluated both frozen logs on the same 70 held-out tasks, all run anew (in Table A18 the smaller log solved 59 / 64). Each cell gives GPT-4.1-mini / GPT-4.1. The last two rows also expose episode outcomes in the file listing.

LogTasks solved out of 70Search cost (¢/task)Files opened per task
70 episodes55 / 637.6 / 7.03.3 / 3.3
1,000 episodes54 / 6410.6 / 12.21.2 / 1.8
70 episodes, outcome labels in the file listing59 / 638.2 / 8.12.4 / 2.9
1,000 episodes, outcome labels in the file listing60 / 6716.8 / 17.21.1 / 1.2

With the original listing format, the larger log mainly increased search cost, while solved-task counts remained similar. The agent still opened only a few episode files, and none of the searches reached the round limit or needed context compaction. This supports selective retrieval from a larger log without absorbing the whole history. With the outcome labels, success was not lower at 1,000 episodes, and a search cost about twice as much. This test measures retrieval from a larger frozen store, while the online-growth analysis covers 140 episodes. We will add these results to the revision and clarify the distinction between store size and online horizon, together with the model dependence of the log-versus-curation comparison.

W2 & Q2: Trees, links, and tags

We agree that a single tree cannot capture every semantic relation. Path references let a file have one location while remaining connected to several topics. In your crawler example, the Python and networking notes can both point to the error log, wherever it is stored. The memories use path references for this purpose (233 in one memory; Sec. 4.1.1), and search can follow them or search across the whole store. Related information therefore need not share one parent folder to remain accessible.

The default tools mirror the six operations of the industry-default memory tool (Sec. 2.2), so they include no link or tag operation; filesystem links were available through the shell harness, although no agent created one in 14,523 recorded shell calls. Dedicated tag indexing remains an untested alternative. Proximity in the tree is a goal for organizing memory, while usefulness is evaluated separately through task performance and cost.

To test how much retrieval depends on the chosen hierarchy, we flattened or randomly reassigned files in three PersonaMem 128k memories without changing their contents (path references were updated to the new paths). We also repeated search on the original layouts. Cells give correct answers on the same 42 questions.

MemoryOriginal layout, repeated runsFlattenedRandom reassignment
built by GPT-5.4-nano, 122 files29, 293229
built by GPT-5.4, 105 files30, 312929
built with shell tools, 147 files28, 31, 252722, 23 (first layout), 29 (second)

Flattening caused no consistent accuracy decline. For the shell-built memory, the original layout itself scored 25 to 31 over three searches. One random reassignment scored 22 and 23, just below that range; a second scored 29, inside it. Two reassignments cannot separate a harmful layout from run-to-run variation, so we do not claim an effect of the layout on accuracy. The results support evaluating hierarchy as one aid to search, and they do not require it to represent every semantic relation perfectly. We will clarify the role of cross-branch references and add the layout perturbation test to the experimental analysis.

AuthorsOpenReview ↗

Response to Reviewer NRVR (2/3)

W3: Rollback

Rollback is a useful safeguard, and it is outside the scope of this paper: we study how agents organize a file memory and what that organization is worth, not a production-ready memory system. Because the memory is plain files, standard practice such as git version control or write-ahead logging applies as it is. We will note this in the revised paper.

W4 & Q3: Latency and model calls

Thank you for requesting these measurements (Table A17 gives calls per query on all four benchmarks). We agree that latency needs to be considered alongside query cost. We measured both for the full search-agent run on PersonaMem 128k, one question at a time, searching each memory twice on the same 42 questions.

MemorySeconds/query, median (mean)Mean model calls/queryMean tool calls/queryCost (¢/query)
BM25 chunk retrieval5.8 (8.6)2.62.60.9
Verbatim session files, one folder12.3 (12.9)3.64.32.7
Verbatim session files in folders13.7 (15.8)3.94.92.5
Reorganized by an agent (two variants)13.1 (15.2), 15.1 (18.3)5.0, 5.87.2, 8.11.2, 1.3
Curated by an agent15.6 (19.3)5.38.41.7

(The cents are lower than in the paper's tables because more of the input came from the provider's cache in these runs; the order of the memories is the same.)

In separate matched runs with the same chunks and top-3 cutoff, text-embedding-3-large took a median of 7.0 to 7.7 seconds per query, versus 6.4 to 6.7 for BM25, with 2.4 to 2.6 model calls on average.

File-based search was slower than chunk retrieval in these runs and took more model rounds. More rounds can take longer even when they process less text overall, which explains how reorganization can lower query cost without reducing latency. Local tool work took less than 0.1 seconds per query. The paper claims the cost saving of reorganized over verbatim files (Sec. 4.2.1) and makes no claim about speed.

The search also has guidance beyond a top-down traversal: file descriptions, whole-store text matches, and, in the BM25 harness, ranked file retrieval. That harness did not consistently reduce model calls or change accuracy (Tables A7 and A16). The timing results make the tradeoff explicit and help determine whether reorganization's cost savings suit a particular use case. We will report these timings, invocation counts, and the embedding baseline alongside retrieval cost in the revision.

AuthorsOpenReview ↗

Response to Reviewer NRVR (3/3)

Q4: End-to-end token usage over 140 tasks

We agree that the comparison should include all work needed to maintain and use the memory. Table A15 reports the combined management, search, and execution cost for the single 140-task chains. Sec. 4.2.1 also sets the management cost against the search savings. Below is the requested cumulative token breakdown. Role columns give millions of input / output tokens, including cached and uncached input. Totals sum the rounded role counts; dollar costs account for caching and model prices.

Execution agent and memoryManagementSearchExecutionTotal (M tokens)Total (USD)Success (%)
GPT-4.1-mini, no memory0031.8 / 0.432.24.5152.9
GPT-4.1-mini, Episode log013.5 / 0.927.0 / 0.441.812.2173.6
GPT-4.1-mini, curated memory26.3 / 1.25.2 / 0.520.6 / 0.354.117.5276.4
GPT-4.1, no memory0023.0 / 0.423.417.3573.6
GPT-4.1, Episode log011.6 / 0.815.0 / 0.327.718.9388.6
GPT-4.1, curated memory14.3 / 0.87.2 / 0.415.8 / 0.238.723.7784.3

As noted in App. E.3, the retrieval savings did not offset ongoing management over these chains. Reusing a memory changes that calculation because the same management work can serve many later tasks. To test this, we froze the curated memory and log after task 70 and evaluated them on 134 tasks from ALFWorld's unseen split.

Execution agentSolved out of 134 (curated / log / no memory)Saving vs. log (¢/task)Served tasks to break even
GPT-4.1-mini98 / 94 / 663.4 to 7.2159 to 202
GPT-4.1121 / 114 / 1066.7 to 13.473 to 96

Curated memory was cheaper to use under all three cache-accounting schemes: recorded caching, a common cache share, and no caching. The success differences from the log were not statistically significant (a first, interrupted run of the GPT-4.1 curated cell solved 115). Break-even counts the tasks served with the frozen memory until the saving repays the management cost of tasks 1 to 70: first the 70 held-out tasks of Table A18, then these 134. For both agents it falls within these 204 tasks. (The "around task 870" of App. E.3 is a different quantity: it is for a chain in which management goes on at every task.)

The practical distinction is between paying for continuous updates and reusing what has already been built. The paper accounts for the first cost, and this additional test estimates when reuse can repay it. We will add the token breakdown by role and the frozen-memory reuse results to the cost analysis, with a clear distinction between measured costs and break-even estimates.

We hope this answers your questions, and we are happy to clarify anything further.

Dear Reviewer NRVR,

We would like to thank you once more for your detailed review and the questions you raised. Our responses to your comments have been posted, and we would welcome your feedback on them. Should any point need more explanation, we would be glad to provide it. We appreciate the time and effort you have given to our submission.

Best regards,

The Authors

Reviewer_Az7POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies filesystem-based memory for LLM agents, in which memory is a directory tree of markdown files that the agent itself reads, writes, and reorganizes through generic file tools. The paper systematically formalizes three roles of an agentic memory system: (1) a management agent, (2) a search agent, and (3) an execution agent. Through experiments, the paper answers four research questions (RQs) about good memory management and organization.

Critical strengths

This paper raises interesting research questions, and the experiments answer them.

Critical weaknesses

(1) For the paper to claim We present, to our knowledge, the first systematic exploration of filesystem-based memory for LLM agents (line 17-18), it would be nice to cite the file-based agents in MEME https://arxiv.org/abs/2605.12477, StructMemEval https://arxiv.org/abs/2602.11243, and several open-source community works (e.g., Karpathy's llm-wiki https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) as related work, and it should clearly specify the scenario of this paper that distinguishes it from them. The works mentioned above are relatively recent or come from the open-source community, so comparing with them may not be mandatory under the reviewer guidelines. However, it would be good to do so to strengthen the distinctiveness of this paper.

(2) The paper proposes such a memory agent (harness), but the agent itself is straightforward in its composition and behavior. In other words, in my view, for this paper to have a strong contribution, I expect that it should either present a target scenario that is both novel and practical (one newly discovered by this paper), or the proposed agent harness should make a significant contribution. I may not have properly caught the advantages on the agent side, but at the moment, I am not sure which part is new.

(3) The appendix of this paper is too long (more than 100 pages), so it is difficult for the reviewer to grasp all the details. Thus, the reviewer evaluates the paper mainly based on what is covered as important in the main paper.

Questions for authors

What exactly is the scenario that distinguishes this work from MEME's file-based agents, StructMemEval and community patterns such as llm-wiki? The reviewer suggests adding a related-work section that states this difference, and revising the "first systematic exploration" claim relative to it.

Which part of the work do the authors consider new: the target scenario, the agent harness, or the empirical findings? If it is the scenario, what makes it novel and practical beyond the existing benchmarks used? If it is the harness, what does it add beyond the file tools of deployed memory systems?

2 comments

We thank the reviewer for the careful review, and for noting that the paper raises interesting research questions and that the experiments answer them.

W1 & Q1. Related work and scope of the study

Thank you for pointing us to these works. The key difference is the research question. We ask whether organizing a file memory is worth the effort, and what determines its value as the memory grows. The table below makes the scope of this comparison explicit.

WorkMain focusCompares alternative folder layouts with file contents held fixed?Quantifies filesystem layout alongside performance and cost?Tracks filesystem growth across a sequence of completed tasks?
MEMEBenchmarks reasoning about changing and interdependent facts.No. Compares complete memory systems, including MD-flat and Karpathy Wiki.No. Reports answer accuracy and ingestion and inference cost for each system.No. Tests whether systems retain and use facts before and after changes within an episode.
StructMemEvalTests whether agents discover or use task-specific structures, such as ledgers or family trees.No. Compares agents with and without hints about how to represent the task's information.No. Measures task accuracy and reviews the memory structures that agents produce.No. Examines memory updates within individual benchmark problems.
Karpathy's LLM WikiDescribes an agent-maintained, interlinked Markdown wiki.No. Describes a workflow for one wiki design.No. Provides design guidance for maintaining the wiki.No. Describes how the wiki can grow as new sources are added.
Our studyStudies the costs and benefits of organizing a growing file memory.Yes. Foldered sessions moves the original files into folders without editing them. We also compare rewriting and incremental curation.Yes. Measures properties of the folder, file, and heading hierarchy, as well as answer quality, task success, and cost.Yes. Records a snapshot after each task in 140-task chains and measures how memory use and performance change over the chain.

For example, in StructMemEval's family-tree tasks, the tree represents kinship relations. In our study, it is the storage layout the agent chooses. Our contribution is a controlled study of the costs and benefits of organizing file memory and how they change with continued use. We will state this specific contribution in place of the broad "first systematic exploration" claim and add a Related Work section with this comparison.

W2 & Q2. What this study contributes

Of your three options, the new part is the empirical findings, with the controlled study behind them. The scenario, file-based memory, is already used in practice, but its adoption does not establish whether having an agent organize it is worthwhile. Our study tests that assumption. We deliberately use a straightforward harness because the question concerns choices within existing systems, rather than the design of another memory agent.

1. We explain what organizing memory actually buys. A ranking of complete memory systems leaves open why one works better. Our study distinguishes organizing the same information from rewriting it, which can lose information needed later (free reorganization shrank a 612 KB memory to 20 KB; Sec. 4.1.1). This helps decide whether and how to reorganize an existing memory: cheaper retrieval alone is not enough to justify the change.

2. The simple harness is itself worth studying. The tool set is a factor we vary: with the model fixed, we compare the plain file tools, the same tools plus ranked search, and a full shell (Sec. 4.1.3). On the two conversational benchmarks tested, accuracy stayed within run-to-run variation and model calls did not fall consistently (Sec. 4.2.3; Table A7).

3. Better memory depends on who writes it and who uses it. The preferred choice between raw experience and curated guidance can change with the execution agent (Sec. 4.2.1). A stronger management agent can also improve a fixed execution agent by providing better guidance (Table 2). These findings help decide whether to strengthen the management agent or the execution agent.

W3. Appendix length and navigation

The main evidence for the contribution above is already in Secs. 2 to 4. Two further findings come from the appendix's growth and cross-family analyses: the raw log's advantage over no memory rose from about 11 to about 20 percentage points between the first and last 35 ALFWorld tasks at both execution tiers (App. E.3), and no conversational memory variant led accuracy across all benchmarks and the three model families (Table A29). We will bring these findings into the main text and add an appendix guide linking each claim to its supporting results and reproduction details.

We hope this answers your questions, and we are happy to clarify anything further.

Dear Reviewer Az7P,

Thank you again for your careful review and your constructive comments. We have posted our responses to the points you raised, and we would be glad to hear whether they address your concerns. If anything remains unclear, we are happy to clarify it. Thank you for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment

We sincerely thank the three reviewers, the AC, and the SAC for their time and thoughtful consideration. We have responded to every point raised in the reviews and uploaded a revised paper, with changes marked in blue.

Our paper is a controlled study of the costs and benefits of organizing filesystem-based agent memory and of how they change with continued use. We are encouraged that the reviewers noted that it "raises interesting research questions, and the experiments answer them" (Az7P), provides "a rigorous benchmark framework for agentic memory maintenance" (NRVR), and has a ladder of memory variants that is "the right design for the question" (f9ti). Following their suggestions, the main changes are:

  • Search savings without an accuracy gain remain on a larger sample [f9ti W1, Q1]. On eleven further PersonaMem conversations, reorganized memories cost about half as much to search as verbatim files and are no more accurate. Repeated builds on PersonaMem 128k preserve the main structural contrasts between management models, and the accuracy differences between them are now stated as an upper bound on the effect. Revised paper: Table 1; Sections 4.1 and 4.2; Appendix H.1 and H.2.
  • Scattering sections across files increased search cost in all three many-file memories, even though the text was preserved [f9ti Q2; NRVR W2, Q2]. Folder changes alone had no consistent effect on accuracy or cost. Revised paper: Section 4.2.1; Appendix H.3.
  • A frozen 1,000-episode log solves about as many held-out tasks as a 70-episode log, at a higher search cost [NRVR W1 to W4, Q1 to Q4]. The value of curation depends on the agents: a curated variant has the highest observed success in 5 of 6 settings (three model families, two execution agents). We also report latency and call counts with an embedding baseline, tokens per agent role, and frozen-memory reuse, with notes on cross-branch references and rollback. Revised paper: Sections 4.2.1 and 4.3; Appendix H.5 to H.8.
  • The main text now covers all four research questions [Az7P W3; f9ti W1, W2]. Section 4.3 holds the main findings on growth and on two other model families. Over 140-task chains, memory keeps its advantage over no memory, and files already written largely keep their lexical fit. Revised paper: Section 4.3; Appendix H.4.
  • The contribution is restated and set against related work [Az7P W1 to W3, Q1, Q2]. The abstract and Section 1 state the specific contribution above in place of the broader "first systematic exploration" claim. What is new in the work is the empirical findings, with the controlled study behind them; the file tools are deliberately plain. A new Related Work section compares the study with MEME, StructMemEval, and the LLM Wiki, and an appendix guide links each claim to its evidence. Revised paper: Abstract; Sections 1 and 5; appendix guide (page 16).

We thank Reviewer NRVR for reconsidering and raising the rating, and Reviewer f9ti for the follow-up comment and for maintaining a positive rating. We have not yet received Reviewer Az7P's feedback on our response, which answers each point of the review, and we would warmly welcome any thoughts on whether it resolves the concerns raised. We are happy to clarify anything further.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.