acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Filesystem Memory for Long-Horizon Embodied Agents

Abstract

Long-horizon agents need memory to retain and reuse experience across long interactions, but how to represent that memory remains unclear. Retaining complete interaction histories makes relevant experience difficult to find, while compressing them into summaries can discard visual and spatial details needed later. We introduce **MFM**, a multimodal filesystem memory architecture that maintains complementary forms of agent memory. **Episodic memory** preserves observations, actions, and outcomes, while **spatial memory** connects experiences to locations. **Semantic and procedural memory** accumulate knowledge and reusable ways of acting, and **working memory** maintains ongoing plans. MFM uses searchable textual records to link these representations to the underlying multimodal episodes. Agents can locate past experiences through contextual cues and inspect original observations to recover details absent from their textual descriptions. Through consolidation, agents turn experience into higher-level knowledge and procedures while retaining references to the supporting episodes. Across long-horizon Minecraft and computer-use tasks, MFM improves performance across three base models. With GPT-6-Astra, MFM completes ***Mine Diamond from Scratch*** and reaches 16 of 24 milestones on the ***Ender Dragon*** task, compared with 12.7 for the strongest baseline. On CUABench-KiCad, it achieves a mean reward of 0.440, compared with 0.400 for the strongest baseline and 0.267 without persistent memory.

Primary area:
applications to robotics, autonomy, planning
Keywords:
agentic memory, language model agents

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_pCGrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes MFM, an external memory for long-horizon multimodal agents laid out as a filesystem. The harness writes immutable records of each procedure-call "episode" (frames, state, action, outcome). It also writes a JSONL index of cues (steps, x/y/z, action kind, tags, outcome summary) that point back to those records, plus a grid of traversed cells. The agent writes revisable notes, a guide, executable procedures, and a working plan, each citing its supporting episodes. Retrieval first searches the cues and then inspects the referenced frames (crop, contact sheet, then-vs-now comparison), and a periodic consolidation step rewrites the agent's memory. The authors map these pieces onto episodic, spatial, semantic, procedural, and working memory. Experiments cover two MCU Minecraft tasks and 25 CUABench-KiCad tasks with three frontier models, against a no-memory agent and reimplementations of PRO-LONG and VISTA. With GPT-6-Astra, MFM reaches 10/10 Diamond milestones and 16/24 Dragon milestones (strongest baseline: 12.7), and 0.440 vs. 0.400 KiCad reward.

Critical strengths
  1. The experimental comparison is controlled and valid. All methods share the same 28 closed-loop atoms, 12 KB brief, plan cap, and [RECENT]/[MAP] block. The comparison spans three models, two Minecraft tasks, and a GUI domain. On Ender Dragon with GPT-6-Astra, MFM reaches 16.0±0.0 milestones vs. 12.7±0.6 for VISTA, and Fig. 7 shows the baselines plateauing. In the one-run cost comparison (Table 8), MFM is also the cheapest.
Critical weaknesses
  1. The ablation does not isolate components. Table 4 uses only GPT-5.6-Sol on Minecraft, the setting with the smallest margin. Every single-component ablation falls below VISTA (Diamond 8.0–8.7 vs. 9.3). That includes "No agent procedures" and "No consolidation", which still keep the index and matched inspection. Spatial and working memory, two of the five headline memory forms, are never ablated. Ablations on GPT-6-Astra/Dragon and on KiCad are needed.

  2. The novelty is limited; there are missing baselines and related work. The ingredients are established:

    • retrieval over a memory stream plus reflection (Generative Agents, Park et al., 2023);
    • agent-managed tiered memory (MemGPT, Packer et al., 2023);
    • skill libraries (Voyager);
    • multimodal experience pools (JARVIS-1, Optimus-1);
    • place-event memory (MrSteve);
    • screenshot-based episodic memory for computer control (Cradle, Tan et al., 2024).
Questions for authors
  1. Could the authors report the ablations with GPT-6-Astra on Dragon? Why does every GPT-5.6-Sol ablation fall below VISTA?

  2. Do the reimplemented PRO-LONG and VISTA keep their memory across deaths ("lives")?

  3. For each model, what scaffold, context policy, and reasoning settings were used? What triggers consolidation, and how are episode boundaries defined?

6 comments
AuthorsOpenReview ↗

Response to Reviewer pCGr (1/3)

[W1 & Q1] Ablations with GPT-6-Astra, on KiCad, and of Spatial and Working Memory, and Why MFM GPT-5.6-Sol Ablations Fall Below VISTA (GPT-5.6-Sol). We thank the reviewer for these questions. As suggested by the reviewer, we extended the ablations along each listed axis (Tables 4 and 7). All ablations now run on both GPT-5.6-Sol and GPT-6-Astra and on both Mine Diamond and Ender Dragon. We added variants that remove spatial memory (no map, grid, or radius queries) and working memory (working.md deleted before every call), as well as frozen-knowledge, no-evidence-link, and caption-only variants. With GPT-6-Astra on Ender Dragon, where full MFM reaches 16.0, the ablations range from 9.3 (no consolidation, or no index) to 14.0 (no working memory), and removing spatial memory gives 12.0 ± 2.9. On KiCad with GPT-6-Astra, removing consolidation or history images lowers reward from 0.440 to 0.427, and removing cross-task memory lowers it to 0.413. As for why every GPT-5.6-Sol Diamond ablation falls below VISTA: full MFM leads VISTA there by only 0.4 milestones (9.7 vs. 9.3). The last milestone, mining a diamond, requires descending about 55 blocks and branch-mining within 12,000 steps. MFM shows the model only the current frame and retrieves past frames on demand. VISTA keeps a 4 MB buffer of recent frames in context and calls the model 3–4× as often. Each MFM component makes up for part of that difference, so removing any one of them leaves the agent stalled before reaching diamond. On the longer Ender Dragon task with GPT-5.6-Sol, the picture changes: four ablations match VISTA (9.0) and two exceed it. One of those results runs against our expectations, and we report it as such: deleting working memory with GPT-5.6-Sol on Ender Dragon gives 11.5 ± 3.5, above full MFM's 10.7 ± 0.6, although with high variance.

Variant (Tables 2, 4, 7)Sol Diamond (/10)Sol Dragon (/24)Astra Diamond (/10)Astra Dragon (/24)Isolated Role
Full MFM (Ours)9.7 ± 0.610.7 ± 0.610.0 ± 0.016.0 ± 0.0All components active
MFM: No working memory6.7 ± 1.211.5 ± 3.59.3 ± 0.614.0 ± 1.0Deletes working.md
VISTA [1]9.3 ± 0.69.0 ± 1.09.7 ± 0.612.7 ± 0.64 MB frames + WORKING/GUIDE
MFM: No spatial memory7.0 ± 1.79.0 ± 0.08.0 ± 1.712.0 ± 2.9No [MAP], grid.jsonl, --near
MFM: Frozen knowledge7.7 ± 1.510.0 ± 0.09.0 ± 0.011.5 ± 2.1Keeps working.md; freezes notes/scripts
MFM: No procedures8.7 ± 0.69.0 ± 1.08.7 ± 0.610.7 ± 1.2Disables procedures/
MFM: Flat memory8.7 ± 0.68.7 ± 0.69.3 ± 0.69.3 ± 0.6Removes index.jsonl
MFM: No consolidation8.3 ± 0.68.3 ± 0.68.7 ± 0.69.3 ± 0.6Disables mine/ writes
MFM: No evidence links8.3 ± 0.69.0 ± 0.08.7 ± 0.69.7 ± 0.6Strips ep_NNNNNN links from mine/
MFM: No visual obs.8.0 ± 1.08.3 ± 0.68.7 ± 0.69.7 ± 0.6No PNGs (matches PRO-LONG 8.0 on Sol)
MFM: Caption-only8.0 ± 1.79.0 ± 0.09.0 ± 0.09.7 ± 0.660–150w captions at write time
PRO-LONG [2]8.0 ± 1.08.3 ± 0.69.3 ± 0.611.0 ± 1.0Text logs.txt without frames

We have updated Sec. 4.3 and Appendix D.3 to add further GPT-5.6-Sol, GPT-6-Astra, and KiCad ablations and to explain why removing any single component drops MFM below VISTA on Mine Diamond.

References

  • [1] Han et al. VISTA: A Visual Harness for Reasoning in an Interactive World. arXiv, 2026.
  • [2] Fox et al. PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning. arXiv, 2026.
  • [3] Tan et al. Cradle: Empowering Foundation Agents Towards General Computer Control. arXiv, 2024.
  • [4] Wang et al. JARVIS-1: Open-World Multi-Task Agents with Memory-Augmented Multimodal Language Models. IEEE TPAMI, 2025.
  • [5] Li et al. Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks. NeurIPS, 2024.
  • [6] Packer et al. MemGPT: Towards LLMs as Operating Systems. arXiv, 2023.
  • [7] Park et al. Generative Agents: Interactive Simulacra of Human Behavior. UIST, 2023.
  • [8] Park et al. MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory. ICLR, 2025.
  • [9] Wang et al. Voyager: An Open-Ended Embodied Agent with Large Language Models. TMLR, 2024.
AuthorsOpenReview ↗

Response to Reviewer pCGr (2/3)

[W2] Controlled Comparison Against Six Prior Memory Architectures. We thank the reviewer and agree that each listed ingredient has precedent and that MFM does not introduce them individually; Sec. 2 and Table 1 now include Generative Agents and MemGPT. Our contribution is an embodied-agent memory architecture built on one design decision: raw multimodal episodes are kept unchanged, and everything the agent writes (notes, guides, procedures, plans) cites the episodes it came from. The agent can thus revisit original frames when a visual detail matters later, and check the evidence behind a note or script before revising it. Prior systems depart from this differently: Generative Agents and MemGPT convert experience to text at write time; JARVIS-1, Optimus-1, MrSteve, and Cradle keep frames or places, but nothing the agent writes links back to them; Voyager's skills are detached from their source episodes. As suggested, we ran adapted versions under the same brief, action atoms, and GPT-6-Astra settings (3 seeds; Table 7). On Ender Dragon they reach 6.0–10.0 milestones vs. 16.0 for MFM and 12.7 for VISTA; on KiCad, Cradle reaches 0.373 vs. 0.440. These are our adaptations, not the original full pipelines (Appendix D.3).

Architecture (GPT-6-Astra, 3 Seeds; Dragon / 24 unless noted)Benchmark ScoreMemory Mechanism & Observed Bottleneck
Full MFM (Ours)16.0 ± 0.0Cue index + raw PNGs + grid.jsonl + linked mine/
MFM + Multimodal Embed14.7 ± 1.2Predicate search → bge-small-en-v1.5 + CLIP ViT-B/32 rank fusion; all else identical
JARVIS-1 / Optimus-1 [4, 5]10.0 ± 4.0Plan keyframes + outcomes (bge + CLIP); no editable revision
MemGPT [6]10.0 ± 0.0Core memory + FIFO text summaries + bge
Generative Agents [7]9.0 ± 1.0Recency + importance + bge + reflections; no frames
MrSteve [8]7.3 ± 1.5CLIP place memory; no outcome cues or notes
Voyager [9]6.0 ± 1.7Skill library
Cradle [3] (KiCad, 25 tasks)0.373 ± 0.009Rolling screenshot buffer; trails MFM (0.440), MFM w/o cross-task (0.413), VISTA [1] (0.400)

Table 7 (new in revision). Additional baselines and controls (milestones; Diamond / 10, Dragon / 24; same brief and action atoms; — = not run):

Method / Control VariantSol Dia.Sol Drag.Astra Dia.Astra Drag.
Main-table reference methods (Table 2)
MFM (ours)9.7 ± 0.610.7 ± 0.610.0 ± 0.016.0 ± 0.0
VISTA9.3 ± 0.69.0 ± 1.09.7 ± 0.612.7 ± 0.6
PRO-LONG8.0 ± 1.08.3 ± 0.69.3 ± 0.611.0 ± 1.0
Direct Evaluation4.0 ± 1.06.0 ± 1.06.3 ± 0.66.3 ± 0.6
Adapted prior-paradigm & multimodal-embedding baselines
MFM + Multimodal Embed———14.7 ± 1.2
JARVIS-1 / Optimus-1———10.0 ± 4.0
MemGPT———10.0 ± 0.0
Generative Agents———9.0 ± 1.0
MrSteve———7.3 ± 1.5
Voyager———6.0 ± 1.7
Representation, consolidation & procedure-store controls
MFM w/o working memory6.7 ± 1.211.5 ± 3.59.3 ± 0.614.0 ± 1.0
PRO-LONG + MFM procedures/———13.0 ± 1.7
VISTA + MFM procedures/———12.3 ± 3.2
MFM w/o spatial memory7.0 ± 1.79.0 ± 0.08.0 ± 1.712.0 ± 2.9
MFM w/ frozen knowledge7.7 ± 1.510.0 ± 0.09.0 ± 0.011.5 ± 2.1
MFM w/o evidence links8.3 ± 0.69.0 ± 0.08.7 ± 0.69.7 ± 0.6
MFM w/ write-time captions8.0 ± 1.79.0 ± 0.09.0 ± 0.09.7 ± 0.6

Table 1 (revised; † = new row). MFM vs. prior agent memory systems:

MethodRaw multimodal evidenceIndex with evidence pointersSpatial memoryAgent-writable memoryKnowledge linked to evidenceRetrieval
Generative Agents †✗✗✓✓✓Recency + importance + embedding relevance
MemGPT †✗✗✗✓✗Text, timestamp, and embedding search
Voyager✗✗✗✓ᵃ✗Embedding similarity
JARVIS-1✓ᵇ✗✗✓ᵇ✗Multimodal similarity
MrSteve✓✗✓✗✗Place/event lookup
PRO-LONG✗ᶜ✗✗✓ᵈ✗ᵈRegex / code over log
VISTA✓✗ᵉ✗✓ᵉ✗Inspect by frame ID
MFM (ours)✓✓✓✓✓Cue query + evidence inspection

ᵃ Skill library (code) only. ᵇ Successful trajectories only. ᶜ Text log only. ᵈ Workspace files/notes; structured text log lacks visual frames/links (revised). ᵉ Frame IDs without a searchable index; free-form notes.

We updated the Abstract, Sections 1–2 and 4.2, and Appendix D.3 with these comparisons.

AuthorsOpenReview ↗

Response to Reviewer pCGr (3/3)

[Q2] Do the reimplemented PRO-LONG and VISTA keep their memory across deaths? We thank the reviewer for these questions. Yes. In our reimplementations, PRO-LONG, VISTA, and MFM all keep 100% of their persistent memory across deaths within a run. PRO-LONG keeps its full logs.txt and workspace; VISTA keeps WORKING.md, GUIDE.md, and its stored frames; and MFM keeps its episodes, index, and agent-written files. The same holds across the 25 sequential KiCad tasks. The main Minecraft benchmarks are single runs that start with no prior memory, so retention there applies to deaths within the run. We now state this in Sec. 4.1 and Table 6.

[Q3] Scaffolds, context policy, reasoning settings, consolidation triggers, and episode boundaries. We added these details to Sections 3.4 and 4.1 and Appendix D.3. All methods use high reasoning effort with every model. The only exceptions are internal helper calls inside the Generative Agents and MemGPT baselines (importance scoring and summarization), which use low effort. MFM and PRO-LONG run in persistent Codex CLI (v0.153.4, bwrap sandbox) or Claude Code CLI sessions, with automatic context compaction at 100k tokens. VISTA runs in its tool-server loop or in Claude Code, with its own checkpoint once stored images reach 4 MB. The Direct baseline makes stateless single API calls. At each call, MFM, PRO-LONG, and VISTA all receive the current frame plus state, recent-plan, and map blocks. In addition, MFM receives its working-memory file, PRO-LONG the tail of logs.txt, and VISTA its WORKING and GUIDE files and inspect tool. Direct receives only the frame and state. In Minecraft, an episode is one plan entry: one primitive action with its repeat count, or one procedure call. The harness saves the frame and state before the action at its first step, and saves the action, outcome, and index entry at its last step. In KiCad, an episode is one batch of GUI actions. Consolidation works as described in our reply to Reviewer JhJR. The agent maintains working.md continuously across compactions. It writes to durable files only on four events: a predicted outcome is confirmed, a rule or coordinate is verified, a safety reflex interrupts an action, or the agent takes damage or dies. At run boundaries, the harness indexes each finished run under previous/.

MethodScaffold & Session PolicyContext Given at Each Model CallAcross Deaths
MFMCodex CLI v0.153.4 (bwrap) / Claude Code CLI; persistent, 100k auto-compactFrame + [STATE] + [RECENT] + [MAP] + [WORKING]Keeps episodes/, index.jsonl, mine/
PRO-LONG [2]Codex CLI v0.153.4 (bwrap) / Claude Code CLI; persistent, 100k auto-compactFrame + [STATE] + [RECENT] + [MAP] + logs.txt tailKeeps full logs.txt & workspace
VISTA [1]Tool-server loop / Claude Code CLI; persistent, 4 MB image auto-checkpointFrame + [STATE] + [RECENT] + [MAP] + WORKING/GUIDE + inspectKeeps WORKING.md, GUIDE.md, frames
DirectStateless single API calls (GPT-6-Astra / GPT-5.6-Sol / Claude Opus 5); no persistent sessionCurrent frame + [STATE] only (no [RECENT], [MAP], or files)None (memory-free baseline)

Table 6 (revised paper). Configuration of each method on Minecraft tasks:

CapabilityDirectPRO-LONGVISTAMFM
Standing brief12 KB12 KB12 KB12 KB
[RECENT] / [MAP] in observation blockNoYesYesYes
Episode store and query toolNoStructured per-action log (logs.txt) via regex/code; no visual episode store (revised)Archived frames via inspectYes
Write access to own files; procedure executionYesYesYesYes
Agent-written procedures (observed)0004 written / 14 calls (Diamond; 12 across 3 Dragon seeds)
Plan cap400 steps400 steps400 steps400 steps
Images per callCurrent frameCurrent frameCurrent + inspected framesCurrent + inspected frames

We have updated Sections 3.4 and 4.1 and Appendices B and D.3 to document the scaffolds, reasoning settings, context compaction, the context given at each model call, memory retention across deaths, episode boundaries, and when memory is consolidated.

Reviewer_CSfuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This project proposes Multimodal Filesystem Memory (MFM), a memory architecture for long-horizon embodied agents. MFM organizes episodic, spatial, semantic, procedural, and working memory within a unified filesystem. It preserves raw multimodal observations, maintains searchable indices with references to original evidence, and supports memory consolidation to update knowledge, procedures, and ongoing plans. The authors evaluate MFM on Minecraft and CUABench-KiCad tasks, reporting improvements over memory-free agents and existing memory-augmented baselines.

Critical strengths

Practical motivation. The paper addresses an important challenge in long-horizon agents: retaining and retrieving relevant historical information without losing critical visual or spatial details. The proposed architecture provides a practical solution to this problem.

Promising empirical results. MFM demonstrates generally consistent improvements across three foundation models and two interactive environments. In particular, it achieves 16/24 milestones on the Minecraft Ender Dragon task with GPT-6-Astra, compared with 12.7 for the strongest baseline.

Comprehensive system implementation. The integration of multimodal evidence retrieval, spatial indexing, memory consolidation, and reusable procedures is reasonably well designed. The qualitative examples also illustrate how historical observations can support subsequent decisions.

Critical weaknesses

Limited methodological novelty. My primary concern is that the proposed architecture largely combines established mechanisms rather than introducing a fundamentally new memory method. Preserving original records, constructing compact searchable indices, linking retrieved representations to source evidence, and periodically updating higher-level knowledge have been extensively investigated in retrieval-augmented and agent memory systems. Although MFM adapts these mechanisms to embodied agents through spatial and action-related metadata, the paper does not sufficiently clarify its fundamental methodological contribution beyond this integration.

Insufficient novelty in memory evolution. The proposed Self-Evolving Consolidation mainly involves LLM-based experience summarization, knowledge revision, and reusable procedure generation. These operations closely resemble existing reflection-based and self-evolving agent approaches. The paper does not introduce a clearly differentiated memory update mechanism or systematically examine the reliability of accumulated knowledge.

Gap between research claims and experimental evaluation. The experiments primarily measure downstream task performance rather than directly evaluating the proposed memory capabilities. Important properties such as retrieval accuracy, evidence localization, memory scalability, and knowledge revision reliability are not systematically assessed. Moreover, the ablations demonstrate the usefulness of individual components but do not establish their superiority over established alternatives.

Insufficiently controlled comparisons. The baselines differ in their persistent memory organization and support for reusable procedures. Consequently, it remains unclear whether the improvements arise from the proposed memory architecture itself or from combining additional capabilities.

Questions for authors

What is the fundamental methodological novelty of MFM beyond integrating established memory mechanisms into embodied agents?

How would MFM compare against a conventional multimodal retrieval system equipped with equivalent indexing and consolidation capabilities?

Can the authors provide direct evaluations of retrieval quality, memory scalability, and knowledge revision reliability?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CSfu (1/4)

[W1 & Q1] Methodological Contribution Beyond Prior Memory Mechanisms. We thank the reviewer for this question. We agree that retrieval over a memory stream, reflection, skill libraries, and multimodal experience stores are established, and the revision states plainly that MFM does not introduce these mechanisms individually. Our fundamental methodological novelty is a systematic way to store and organize the past multimodal memory of a long-horizon embodied agent. Following models of human memory (Tulving, 1972; Squire, 2004), we decompose memory into a file hierarchy: episodic memory records each step's frames, states, actions, and outcomes; spatial memory (grid.jsonl) ties them to locations; semantic memory (notes.md, GUIDE.md) holds facts; procedural memory (procedures/) holds reusable procedures; and working memory (working.md) holds the current plan. An index of what/where/when cues (index.jsonl) addresses the episodes, and every agent-written file cites the episodes it came from. This decomposition gives two capabilities that matter for long-horizon embodied tasks. First, episodic memory keeps multimodal detail. A visual detail often turns out to matter only long after it was recorded, so converting observations to text at write time can lose it. MFM keeps the raw frames unchanged, finds episodes through compact cues, and inspects the original frames when a question arises. Second, the agent can correct its memory. Knowledge and procedures derived from experience are often wrong at first. Because each note or script cites its source episodes, the agent can return to the observations, actions, and outcomes behind it to revise or retire it; e.g., re-inspecting the frame behind a "sheep" note reveals a wolf, so the agent corrects the note and its plan while keeping the episode (Sec. 3.4). Rather than argue novelty from performance alone, we test each design choice against an established alternative under the same harness. On GPT-6-Astra Ender Dragon (16.0 for full MFM), storing captions instead of frames gives 9.7, and removing the episode references from agent-written files also gives 9.7. Replacing MFM's retrieval with multimodal embedding search gives 14.7. Adapted versions of MemGPT, Generative Agents, JARVIS-1/Optimus-1, MrSteve, and Voyager, run with the same brief and action interface, reach 6.0–10.0 milestones, and Cradle reaches 0.373 on KiCad vs. 0.440 for MFM. We added Generative Agents and MemGPT to Table 1 and expanded Sec. 2 to make these distinctions explicit.

Method / Control Variant (Ender Dragon, 3 Seeds)Visual Evidence & AddressingMilestones (/ 24)
Full MFMLossless PNGs + index.jsonl + linked mine/16.0 ± 0.0
JARVIS-1 / Optimus-1 [3, 4]Plan keyframes + bge-small-en-v1.5 / CLIP ViT-B/32 (no editable notes)10.0 ± 4.0
MemGPT [1]Core memory + FIFO text summaries + bge10.0 ± 0.0
MFM: Caption-only memory60–150 word GPT-6-Astra captions at write time (no PNGs)9.7 ± 0.6
MFM: No visual observationsStructured text state & outcomes only (no PNGs)9.7 ± 0.6
MFM: Flat memoryLossless PNGs without index.jsonl9.3 ± 0.6
Generative Agents [2]Recency + importance + bge + reflections9.0 ± 1.0
MrSteve [5]CLIP place memory (no outcome cues or notes)7.3 ± 1.5
Voyager [7]Skill library (wrote very few verified skills from raw pixels)6.0 ± 1.7

Table 1 (revised paper; † = row added in revision). MFM versus prior agent memory systems:

MethodRaw multimodal evidenceIndex with evidence pointersSpatial memoryAgent-writable memoryKnowledge linked to evidenceRetrieval
Generative Agents †✗✗✓✓✓Recency + importance + embedding relevance
MemGPT †✗✗✗✓✗Text, timestamp, and embedding search
Voyager✗✗✗✓ᵃ✗Embedding similarity
JARVIS-1✓ᵇ✗✗✓ᵇ✗Multimodal similarity
MrSteve✓✗✓✗✗Place/event lookup
PRO-LONG✗ᶜ✗✗✓ᵈ✗ᵈRegex / code over log
VISTA✓✗ᵉ✗✓ᵉ✗Inspect by frame ID
MFM (ours)✓✓✓✓✓Cue query + evidence inspection

ᵃ Skill library (code) only. ᵇ Successful trajectories only. ᶜ Text log only. ᵈ Workspace files/notes; structured text log lacks visual frames/links (revised). ᵉ Frame IDs without a searchable index; free-form notes.

We have updated the Abstract, Sections 1–2, 3.3–3.4, and 4.2–4.3, and Appendices B and D.3 to clarify how MFM relates to prior memory systems and to add Cradle on KiCad and the comparison with six additional prior systems.

AuthorsOpenReview ↗

Response to Reviewer CSfu (2/4)

[W2 & Q3] Episode-Linked Consolidation, Executable Procedures, and Aggregate Revision Reliability. We thank the reviewer for this question. The individual operations (summarizing experience, revising notes, writing procedures) do resemble reflection-based and self-evolving agents such as Reflexion, A-MEM, Voyager, and MineEvolve. The difference lies in what the updates are tied to and when they happen. Those systems revise text or code that is detached from the original observations. In MFM, every note and procedure cites the episodes that support it. A revision can therefore be checked against the stored frames and outcomes, and a failing script can be debugged by finding and viewing the episodes in which it failed. Updates are also triggered by four concrete events rather than run on a fixed schedule: a predicted outcome is confirmed, a rule or coordinate is verified, a safety reflex fires, or the agent takes damage or dies. With GPT-6-Astra on Dragon, removing the episode references while keeping everything else lowers the score from 16.0 to 9.7, and freezing knowledge while keeping working memory lowers it to 11.5. As suggested by the reviewer, we assessed reliability by auditing the memory at the end of 5 Minecraft runs against the harness logs (Appendix A.4.3). All 46 claims that the logs can confirm or refute (coordinates, game rules, and what a procedure or reflex does) were correct. We excluded 20 lines, such as open subgoals and strategy tips, that logs cannot check. All 5 cases in which stored memory contradicted a later observation were resolved. Three were resolved within the run; for example, a pond recorded to the east actually lay north and was corrected after comparing the old and current frames. Two were resolved in the next life; for example, the agent retired a script after inspecting the lava death it caused. The audit is small, so we present it as evidence that revisions are grounded in observations, not as a guarantee of reliability.

Episode-linked consolidation vs. text reflection & working state (Tables 4, 7):

Variantworking.mdDurable notes.md, GUIDE.md, procedures/Diamond (/ 10)Dragon (/ 24)
Full MFMActiveEpisode-linked10.0 ± 0.016.0 ± 0.0
No working memoryDeletedEpisode-linked9.3 ± 0.614.0 ± 1.0
Frozen knowledgeActiveFrozen / removed9.0 ± 0.011.5 ± 2.1
No consolidationDisabledDisabled8.7 ± 0.69.3 ± 0.6

Correctness, revision, and reuse of stored notes and scripts (Sec. 5.1, Appendix A.4.3, Tables 3, 5–8):

EvaluationWhat Was CheckedHow It Was CheckedResult
Memory audit (5 runs: 2 Dragon, 3 Diamond; Appendix A.4.3)46 checkable claims (22 notes, 20 scripts, 4 reflexes) and 5 conflictsEnd-of-run mine/ files compared against the run logs; 20 uncheckable lines (open subgoals, strategy tips) excluded46/46 claims correct; 5/5 conflicts resolved
Revision across 3 Dragon runs6 note edits; 12 scripts written, 6 revised in place; 1 inherited script retiredPredicted vs. observed outcomes ([EXPECTED] vs. [OBSERVED]), plus milestone and health logs16.0 ± 0.0 milestones vs. 10.7 ± 1.2 without procedures; bridge_north.py crossed lava 9/9 times with no damage
Transfer to new tasks (Table 5: 3 tasks × 5 seeds)Only mine/procedures/*.py copied into a fresh world (no notes, maps, or episodes)Each script checks its own preconditions before running (Appendix C); task evaluators15/15 runs solved with 30%, 7%, and 26% fewer steps (Shield, Climb, Cook)
CUABench-KiCad (Table 3: 25 tasks × 3 seeds)157 task notes and 40 recurring GUI states; no scriptsAgent checks the schematic and runs ERC before saving (Figs. 13–14); netlist comparison0.440 ± 0.013 reward vs. 0.413 without cross-task memory and 0.400 for VISTA

We have updated Sections 3.4, 4.3, and 5.1 and Appendices A.4.3, B, and D.3 to separate the No working memory and Frozen knowledge ablations and to describe the memory audit and repairs.

AuthorsOpenReview ↗

Response to Reviewer CSfu (3/4)

[W4] Controlled Workspace Organization and Procedure-Store Comparisons. We thank the reviewer for this question. All methods share the same 12 KB task brief, 28 closed-loop action atoms, 400-step plan cap, per-call state, recent-plan, and map blocks, and reasoning setting. All of them keep their memory across deaths and have the same shell, file-writing, and Python execution permissions (Table 6). As the reviewer notes, the original baselines differ from MFM in memory organization and procedure support, because their published designs specify those differences. To separate these factors, as suggested by the reviewer, we added controls in which PRO-LONG and VISTA receive MFM's exact procedures folder, reflexes folder, consolidation prompt, and triggers. With GPT-6-Astra on Dragon, they reach 13.0 ± 1.7 and 12.3 ± 3.2, compared with 16.0 ± 0.0 for MFM, so procedure support alone does not explain the gap. The reverse controls point the same way: MFM variants that keep the procedure store but lose either the index (9.3) or the frames (9.7) score below MFM without procedures (10.7). On KiCad, no method writes scripts, and MFM still leads: 0.440, or 0.413 even without cross-task memory, compared with 0.400 for VISTA, 0.375 for PRO-LONG, and 0.373 for Cradle.

Method / Controlled Variant (GPT-6-Astra)procedures/ Storeindex.jsonl CuesDragon (/ 24)Observed Procedure Synthesis
PRO-LONG [11]Shared shellStructured logs.txt (no PNGs)11.0 ± 1.00 scripts written
PRO-LONG + MFM procedures/YesStructured logs.txt (no PNGs)13.0 ± 1.7Scripts from text logs; no frames to verify terrain
VISTA [12]Shared shellNo (frame IDs)12.7 ± 0.60 scripts written
VISTA + MFM procedures/YesNo (frame IDs)12.3 ± 3.2Frame IDs fail to debug routines
MFM: No agent proceduresDisabledYes10.7 ± 1.2GPT-5.6-Sol Dragon: 9.0 ± 1.0 (matches VISTA 9.0)
MFM: No visual observationsYesYes (text only, no PNGs)9.7 ± 0.6Keeps procedures/, loses >6 milestones
MFM: Flat memoryYesNo (flat log + PNGs)9.3 ± 0.6Keeps procedures/, loses >6 milestones
Full MFMYesYes + PNGs16.0 ± 0.0Verified synthesis & reuse

Table 6 (revised paper). Configuration of each method on Minecraft tasks:

CapabilityDirectPRO-LONGVISTAMFM
Standing brief12 KB12 KB12 KB12 KB
[RECENT] / [MAP] in observation blockNoYesYesYes
Episode store and query toolNoStructured per-action log (logs.txt) via regex/code; no visual episode store (revised)Archived frames via inspectYes
Write access to own files; procedure executionYesYesYesYes
Agent-written procedures (observed)0004 written / 14 calls (Diamond; 12 across 3 Dragon seeds)
Plan cap400 steps400 steps400 steps400 steps
Images per callCurrent frameCurrent frameCurrent + inspected framesCurrent + inspected frames

Table 3 (revised paper; † = row added in revision). Mean reward on CUABench-KiCad (25 tasks; mean ± std over 3 seeds):

MethodGPT-5.6-SolClaude Opus 5GPT-6-Astra
Direct Evaluation0.240 ± 0.0260.240 ± 0.0230.267 ± 0.021
Cradle †——0.373 ± 0.009
PRO-LONG0.306 ± 0.0140.300 ± 0.0160.375 ± 0.012
VISTA0.400 ± 0.0110.373 ± 0.0130.400 ± 0.010
MFM (no cross-task memory)0.400 ± 0.0090.375 ± 0.0100.413 ± 0.009
MFM (ours)0.413 ± 0.0090.389 ± 0.0100.440 ± 0.013

We have updated Sections 2, 4.1, and 4.3 and Appendix D.3 to clarify that all methods share the same file-writing and Python permissions and to add the two procedure-store controls.

References

  • [1] Packer et al. MemGPT. arXiv, 2023.
  • [2] Park et al. Generative Agents. UIST, 2023.
  • [3] Wang et al. JARVIS-1. TPAMI, 2025.
  • [4] Li et al. Optimus-1. NeurIPS, 2024.
  • [5] Park et al. MrSteve. ICLR, 2025.
  • [7] Wang et al. Voyager. TMLR, 2024.
  • [11] Fox et al. PRO-LONG. arXiv, 2026.
  • [12] Han et al. VISTA. arXiv, 2026.
Reviewer_cnUWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MFM, a multimodal filesystem memory architecture for long-horizon embodied agents. It implements episodic, spatial, semantic, procedural, and working memory through preserved multimodal episodes, an associative index, and agent-written files. Agents search contextual cues, inspect original observations, and consolidate experience by updating knowledge, procedures, and working memory while retaining references to supporting evidence. Experiments on Minecraft and CUABench-KiCad report improvements across three base models, alongside component ablations and examples of cross-task reuse. The main contribution is the integration of memory organization, retrieval, and revision within a shared filesystem.

Critical strengths
  • Separating preserved episodes from revisable agent-written memory is a useful design choice. References to original observations provide a concrete basis for revisiting uncertain interpretations, and the visual examples illustrate perceptual correction and recovery of spatial context.

  • With GPT-6-Astra, MFM reaches 16 of 24 Ender Dragon milestones versus 12.7 for VISTA, and achieves a KiCad mean reward of 0.440 versus 0.400, providing meaningful evidence of practical benefit.

  • The evaluation explicitly matches the Minecraft action interface and VISTA’s visual inspection capabilities. Four component ablations and a separate cross-task-memory comparison provide useful evidence beyond the selected qualitative examples.

Critical weaknesses
  • The consolidation ablation does not separate learning reusable knowledge from maintaining the agent’s ongoing task state. Consolidation updates semantic, procedural, and working memory together, while the provided prompt emphasizes preserving working memory across context resets. The performance drop therefore does not establish whether knowledge abstraction and evidence-based revision contribute beyond remembering what to do next.

  • This paper describes it as lacking persistent procedure storage and structured action/outcome fields, although the original method supports persistent workspace files and structured records of actions, scores, and resulting observations (PRO-LONG). This also sits uneasily with the stated shared file-writing interface, leaving the attribution of MFM’s procedure creation to associative indexing insufficiently supported.

  • Retrieval depends on agent-selected contextual predicates, while retrieval-lag distributions measure the age of accessed episodes rather than their relevance. The Flat ablation and successful examples support the value of indexing, but leave the reliability of finding the needed visual evidence unmeasured.

  • The parameterized Minecraft routines and procedure ablation support useful skill construction within those tasks. However, the KiCad examples retrieve earlier episodes with nearly identical action sequences, and the cross-task comparison removes all retained memory together. The observed transfer benefit therefore cannot be attributed specifically to learned procedures.

Questions for authors

See the weakness

7 comments
AuthorsOpenReview ↗

Response to Reviewer cnUW (1/4)

[W1] Separating Reusable Knowledge and Procedures from Ongoing Task State in Working Memory. We thank the reviewer for this fair criticism of the original Table 4. As suggested by the reviewer, we added two controls that separate the two (Tables 4 and 7). "No working memory" deletes working.md before every model call, while notes, the guide, and procedures are still written and read. "Frozen knowledge" keeps working.md fully active but freezes notes, the guide, and procedures. The agent can still track what to do next, but it cannot learn or revise reusable knowledge. With GPT-6-Astra, deleting working memory costs relatively little: Diamond drops from 10.0 to 9.3 and Dragon from 16.0 to 14.0, which is still above VISTA's 12.7. Freezing knowledge costs more (9.0 on Diamond, 11.5 on Dragon). On Dragon, it accounts for most of the drop to disabling consolidation entirely (9.3). Frozen knowledge falls below full MFM for both models on both tasks. This directly tests the reviewer's question: keeping track of the current plan does not replace verified coordinates, revised rules, and reusable procedures. With the weaker model, however, the results are mixed, and we report them as such. On GPT-5.6-Sol Diamond, deleting working memory hurts more than freezing knowledge (6.7 vs. 7.7). This suggests that on the shorter 12,000-step task, losing track of the immediate subgoal is the larger bottleneck for that model. On GPT-5.6-Sol Dragon, the no-working-memory variant (11.5 ± 3.5) cannot be distinguished from full MFM (10.7 ± 0.6) given its variance.

Variant (GPT-6-Astra)working.mdDurable notes.md, GUIDE.md, procedures/Diamond (/ 10)Dragon (/ 24)Isolated Change
Full MFMActiveActive10.0 ± 0.016.0 ± 0.0Full architecture
No working memoryDeleted at every model callActive9.3 ± 0.614.0 ± 1.0Deletes working.md (+1.3 vs. VISTA [1] Dragon)
Frozen knowledgeActiveFrozen9.0 ± 0.011.5 ± 2.1Keeps working.md, freezes notes/scripts
No spatial memoryActiveActive (no grid.jsonl/[MAP])8.0 ± 1.712.0 ± 2.9Removes [MAP], grid.jsonl, --near
No agent proceduresActiveNotes active, no scripts8.7 ± 0.610.7 ± 1.2Disables mine/procedures/*.py
No visual / CaptionActiveActive (no PNGs / captions)8.7 / 9.09.7 ± 0.6No PNGs / 60–150w captions
Flat / No consolidationActive / OffNo index / Off9.3 / 8.79.3 ± 0.6Removes index.jsonl / disables mine/

Table 4 (revised paper; † = added in revision, including all GPT-6-Astra results). Ablation of MFM's components (milestones; mean ± std over 3 seeds):

VariantSol Diamond (/ 10)Sol Dragon (/ 24)Astra Diamond (/ 10)Astra Dragon (/ 24)
Full MFM9.7 ± 0.610.7 ± 0.610.0 ± 0.016.0 ± 0.0
No visual obs.8.0 ± 1.08.3 ± 0.68.7 ± 0.69.7 ± 0.6
Flat8.7 ± 0.68.7 ± 0.69.3 ± 0.69.3 ± 0.6
No agent procedures8.7 ± 0.69.0 ± 1.08.7 ± 0.610.7 ± 1.2
No consolidation8.3 ± 0.68.3 ± 0.68.7 ± 0.69.3 ± 0.6
No working memory †6.7 ± 1.211.5 ± 3.59.3 ± 0.614.0 ± 1.0
Frozen knowledge †7.7 ± 1.510.0 ± 0.09.0 ± 0.011.5 ± 2.1
No evidence links †8.3 ± 0.69.0 ± 0.08.7 ± 0.69.7 ± 0.6
No spatial memory †7.0 ± 1.79.0 ± 0.08.0 ± 1.712.0 ± 2.9
Captions-only †8.0 ± 1.79.0 ± 0.09.0 ± 0.09.7 ± 0.6

We have updated Sec. 4.3 and Appendix D.3 to add GPT-6-Astra ablations and to separate the No working memory, Frozen knowledge, and No spatial memory ablations.

AuthorsOpenReview ↗

Response to Reviewer cnUW (2/4)

[W2] PRO-LONG's Structured Records, Shared File-Writing Interface, and Attribution of Procedure Creation to Associative Indexing. We thank the reviewer for catching this error, which we have corrected. Our original Appendix D.3 wrongly stated that PRO-LONG lacks persistent procedure storage and structured action and outcome fields. Both the published PRO-LONG and our reimplementation write structured per-action blocks to logs.txt, covering the plan, the action and its arguments, position and orientation, inventory, verified events, and notes. These blocks can be queried with rg or Python. Every method we compare, including PRO-LONG, also has the same shell, file-writing, and Python execution permissions (Table 6), and we corrected Tables 1 and 6 to reflect this. Three differences remain. First, standard PRO-LONG and VISTA do not provide a procedures folder with a prompt to check observed outcomes against predicted ones. Second, PRO-LONG keeps a chronological log rather than an episode-level index with 3D coordinates and a spatial grid. Third, PRO-LONG stores no frames, and VISTA stores frames but no structured action and outcome records. To test how much of MFM's procedure creation the folder and prompt explain on their own, we gave PRO-LONG and VISTA MFM's exact procedures and reflexes folders, consolidation prompt, and triggers (Table 7, GPT-6-Astra, Dragon). PRO-LONG rose from 11.0 to 13.0 ± 1.7, which shows that its structured records plus the prompt do support writing scripts. VISTA went from 12.7 to 12.3 ± 3.2: it wrote scripts in only 1 of 3 seeds and stalled while debugging them in the other two. Both stay below full MFM (16.0 ± 0.0). The reverse controls point the same way. MFM variants that keep the procedures folder but lose the index (Flat, 9.3) or the frames (No visual, 9.7) score below MFM with procedures disabled (10.7), because scripts that cannot be checked against past terrain misfire. We therefore attribute MFM's procedure gains to structured outcome cues together with linked frames, not to indexing alone, and we revise the text to say so.

Attributing procedure creation and task performance to associative indexing and stored frames (Tables 3–5, 7):

Method / Variant (GPT-6-Astra)procedures/ StoreAction / Outcome RecordPNGsDragon (/ 24)Agent-Written Procedures & Observed Effect
PRO-LONG [2]Shared shellStructured logs.txtNo11.0 ± 1.00 scripts across 3 seeds
PRO-LONG + MFM procedures/YesStructured logs.txtNo13.0 ± 1.7craft.py, excavate.py from text logs; no frames to verify terrain
VISTA [1]Shared shellFrame IDs onlyYes12.7 ± 0.60 scripts across 3 seeds
VISTA + MFM procedures/YesFrame IDs onlyYes12.3 ± 3.2Scripts in only 1 of 3 seeds; stalled debugging routines in the other 2
MFM: No agent proceduresDisabled (0 scripts)index.jsonl + gridYes10.7 ± 1.20 scripts (Sol: 9.0 ± 1.0, matching VISTA 9.0 & beating PRO-LONG 8.3)
MFM: No visual / CaptionYesindex.jsonl + gridNo9.7 ± 0.6Keeps procedures/ & index.jsonl; drops below No procedures without PNGs
MFM: Flat memoryYesFlat logYes9.3 ± 0.6Keeps procedures/ & PNGs; drops below No procedures without index.jsonl
Full MFMYesindex.jsonl + gridYes16.0 ± 0.012 scripts, 6 revised, 1 retired

We have updated Sections 2, 4.3, and 5.1 and Appendix D.3 to correct our description of PRO-LONG's structured logs and to add the procedure-store controls.

AuthorsOpenReview ↗

Response to Reviewer cnUW (3/4)

[W3] Direct Accuracy of Agent-Selected Retrieval Predicates. We thank the reviewer for this point. We agree that retrieval lag measures the age of retrieved episodes, not their relevance, and, as suggested by the reviewer, we added a direct test (Sec. 4.4, Fig. 16). We built 106 ground-truth queries over 5 Minecraft runs, each answered by a specific episode: 37 event, 40 place, 20 time, and 9 visual queries (for example, the earliest frame showing water, lava, iron, or diamond). We score the agent's own predicate queries, with episodes from three other runs added as distractors. MFM reaches 93.4% Hit@1 (99/106), compared with 44.3% for the embedding + LLM reader baseline (GPT-6-Astra reads the top-20 hits and views the top-8 frames) and 2.8% for the embedding-only baseline. The results also show where cue-based retrieval is unreliable, which bears directly on the reviewer's concern about visual evidence. MFM answers all 97 event, place, and time queries correctly, but on queries defined only by visual content it finds the right frame in only 2 of 9 cases (the embedding + LLM reader baseline finds 3). Predicates over coordinates, steps, and outcome tags reliably narrow retrieval to the right moment. When nothing in the index distinguishes the target frame, however, the agent must inspect candidate frames and sometimes picks the wrong one. We report this as a limitation in Sec. 6. When retrieval succeeds, the agent acts on it. Viewing a retrieved frame makes the agent add new navigation targets 2.4–2.5× as often, and 88% of those targets lie within 8 blocks of the retrieved episode (Fig. 17).

Retriever (106 Queries; Hit@1 / Hit@5)Half LifeWhole Life+3 Distractor LivesTop-1 Misses at +3 Lives
MFM Agent (episodes.py + rg + view_image)92.5 / 93.4%91.5 / 93.4%93.4 / 93.4%7 misses: 0 distractor, 0 Event/Place/Time (97/97), 7 visual wrong frames
Embedding + LLM Reader66.0 / 66.0%58.5 / 58.5%44.3 / 45.3%59 misses: 23 wrong-step visits nearby, 18 far, 13 abstain, 4 visual, 1 rank 2–5
Embedding Only (bge + CLIP)19.8 / 34.0%10.4 / 23.6%2.8 / 17.9%103 misses: 63 distractor life, 24 far/visual, 9 rank 2–5, 7 nearby/later

We have updated Sections 4.4 and 6 and Appendices A.4.1–A.4.2 to add the retrieval benchmark, failure cases on visual queries, and an analysis of how retrieved frames change the agent's plans.

Reviewer_JhJROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multimodal Filesystem Memory (MFM), a multimodal filesystem memory architecture for long-horizon embodied agents. MFM integrates five memory types (episodic, spatial, semantic, procedural, and working memory), from which agents can retrieve past experience and turn it into useful information for the current task. Experimental results demonstrate the proposed method’s effectiveness.

Critical strengths

[S1] Practical and well-organized memory architecture.

The proposed framework provides an intuitive and practical way to organize different memory types. In particular, linking searchable textual records to original multimodal observations is a reasonable approach to preserving and retrieving past experiences.

[S2] Consistent experimental improvements.

The results in Tables 2 and 3 show consistent performance improvements across three backbone models and two environments. The ablation studies in Table 4 and qualitative examples further support the effectiveness of the proposed framework.

Critical weaknesses

[W1] Insufficient scalability analysis.

Since this work focuses on long-horizon agents, the analysis should go beyond long memory within a single run to settings where trajectories keep accumulating. Because memory inevitably grows, showing how MFM performs as memory size increases, along with its benefits and limitations, would strengthen the paper.

[W2] Insufficient details of the consolidation mechanism.

Section 3.4 introduces self-evolving consolidation as a key component, but its implementation remains relatively abstract, which makes it difficult to understand. In particular, it is unclear how the experiences to be consolidated are selected. A more detailed explanation would be helpful.

Questions for authors

[Q1] Necessity of raw multimodal evidence.

As Table 4 shows that removing access to historical visual observations reduces performance, I’m curious whether this gain comes from the raw visual evidence itself or simply from retaining more detailed information. Could the authors compare MFM with a text-only memory that stores richer descriptions of visual observations? If it’s computationally impossible, at least the authors' opinion will definitely help.

[Q2] Systematic analysis of memory retrieval.

I acknowledge that the authors provide qualitative examples of how retrieved memories influence agent behavior (Sec. 4.4 & Appendix A). However, the paper would benefit from a more systematic analysis, such as the retrieval success rate, common failure cases and their proportion, and, if possible, how often retrieval changes the agent’s plan or action.

6 comments
AuthorsOpenReview ↗

Response to Reviewer JhJR (1/4)

[W1] Memory Scalability Benefits and Limits. We thank the reviewer for raising scalability. As suggested by the reviewer, we added a multi-run scaling study (Sec. 4.2, Fig. 18, Appendix D.4). We loaded 0 to 10 prior Ender Dragon lives into the archive, which grew it from 4,806 to 47,035 episodes. Over this range, median query latency rose from 399 ms to 899 ms. Hit@1 on 46 ground-truth queries about the current life stayed between 87.0% and 91.3%, compared with 15% for the embedding + LLM reader baseline (GPT-6-Astra reads the top-20 hits and views the top-8 frames) and 0% for the embedding-only baseline at 10 lives. Within a single 36,000-step life, uncached input tokens per call stayed nearly flat from the first to the last tenth of calls (2.6k to 2.9k), because the agent loads only the cues and frames it queries. To test whether more memory actually helps, we also let the agent inherit same-world memory from increasingly long exploration (none, 3k, 6k, and full steps). Ore-mining completion rose from 8.3% (1/12) to 33.3% (2/6), 62.5% (5/8), and 91.7% (11/12) (Fig. 19). On KiCad, carrying memory across 25 sequential tasks raises reward for all three models, e.g., from 0.413 to 0.440 with GPT-6-Astra (Table 3). The study also shows limits. The index is scanned linearly, so the worst-case spatial query, which scans the 42k episodes from the 10 earlier lives, takes 3.4 s, and lossless PNGs make up about 94% of the 2.06 GB archive. Sec. 6 discusses database memory and keyframe deduplication as future directions.

Accumulated MemoryEpisodesDisk SizeMedian episodes.pyLog rgTarget-Life Hit@1 (46 Queries)
0 Prior Lives (36k steps)4,8060.23 GB399 ms20 ms87.0%
1 Prior Life8,9790.41 GB323 ms20 ms89.1%
3 Prior Lives18,3570.83 GB478 ms31 ms91.3%
5 Prior Lives26,3581.19 GB591 ms24 ms89.1%
10 Prior Lives47,0352.06 GB899 ms26 ms87.0% (Embedding + LLM Reader: 15%, Embedding Only: 0%)

Table 3 excerpt (paper). Mean reward on CUABench-KiCad with and without memory carried across the 25 sequential tasks (mean ± std over 3 seeds):

MethodGPT-5.6-SolClaude Opus 5GPT-6-Astra
MFM (no cross-task memory)0.400 ± 0.0090.375 ± 0.0100.413 ± 0.009
MFM (ours)0.413 ± 0.0090.389 ± 0.0100.440 ± 0.013

We have updated Sections 4.2, 5.2, and 6 and Appendix D.4 to add the inherited world-memory experiment, latency, storage, and retrieval results for up to 10 stored lives, and storage costs.

AuthorsOpenReview ↗

Response to Reviewer JhJR (2/4)

[W2] Consolidation Mechanics and Experience Selection Criteria. We thank the reviewer for this question and rewrote Sec. 3.4 to specify it. Experiences enter consolidated memory in two ways. The first is at run boundaries: when a run ends, the harness files it into a fixed structure using fixed rules, with no model judgment involved. In Minecraft, at the end of a life it mounts the run under previous/ and lists its routines in previous/scripts.md. At the start of the next life, the agent inspects the failure or death episodes, updates its guide, and keeps, edits, or retires scripts. In KiCad, at the end of a task the harness groups episodes by the GUI state they were recorded in and writes the states seen at least twice (keeps only the 40 most-frequent recurring GUI states) to previous/situations.md. The second is during a run, where the agent writes to its durable files (notes, guide, procedures, reflexes) only on four events: (i) an observed outcome matches the outcome predicted before acting, and the agent distills the sequence into a procedure; (ii) a rule or coordinate is verified, and the agent records it as a note; (iii) a safety reflex interrupts an action, and the agent updates the reflex or hazard note; (iv) unexpected damage or death occurs, and the agent inspects the surrounding frames to revise its plans or routines. Every write cites the IDs of its supporting episodes. In practice, writes are sparse. Across 3 Dragon seeds (14,420 episodes), MFM made 6 note edits and wrote 12 scripts, of which it revised 6 and retired 1. Over the 25-task KiCad run (927 episodes), it wrote 157 notes and indexed 40 recurring GUI states.

DomainEpisode PoolSelection RuleNotes WrittenProcedures / Groups
Ender Dragon [1]14,420 (3 seeds)4 execution events + life boundary612 written, 6 revised, 1 retired
CUABench-KiCad [2]927 (25-task run)GUI states seen ≥2×\ge 2\times (top 40) + task end15740 situation groups indexed

We have updated Sec. 3.4 and Appendices B and D.3 to specify how past runs are indexed, the consolidation triggers, and how often memory is written.

AuthorsOpenReview ↗

Response to Reviewer JhJR (3/4)

[Q1] Necessity of Raw Visual Evidence vs. Richer Textual Descriptions. We thank the reviewer for this question. As suggested by the reviewer, we ran this comparison (Table 7). We kept the index, map, consolidation, and procedures unchanged and deleted the PNGs. In their place, we stored a 60–150 word text description of every frame, written by GPT-6-Astra when the frame was recorded and searchable as text. The descriptions cover terrain, block distances, ores, liquids, mobs, lighting, and the HUD. With GPT-6-Astra, this caption-only memory scored 9.0 ± 0.0 on Mine Diamond and 9.7 ± 0.6 on Ender Dragon. Full MFM scored 10.0 ± 0.0 and 16.0 ± 0.0, and removing visual observations entirely scored 8.7 ± 0.6 and 9.7 ± 0.6, about the same as captions. GPT-5.6-Sol shows the same pattern (8.0 and 9.0 with captions, vs. 9.7 and 10.7 for full MFM). We conclude that the gain comes from the frames themselves, not just from retaining more detail, for two reasons. First, when the caption is written, the captioner cannot know which detail will matter thousands of steps later. A description such as "dark stone tunnel with torch on the left" can omit a small ore patch on the ceiling or the shape of a ledge. Second, a stored frame can be inspected again for details nobody wrote down. The agent crops and enlarges regions to tell ore from gravel, and it builds side-by-side comparisons of an archived view and the current view. Fixed text supports neither.

Memory Variant (GPT-6-Astra, 3 Seeds)Stored Visual EvidenceDiamond (/ 10)Dragon (/ 24)
Full MFMLossless PNGs + crop & --then-now10.0 ± 0.016.0 ± 0.0
VISTA [3] / PRO-LONG [4]Frames by ID / structured logs.txt9.7 ± 0.6 / 9.3 ± 0.612.7 ± 0.6 / 11.0 ± 1.0
MFM: Caption-only memory60–150 word GPT-6-Astra descriptions (no PNGs)9.0 ± 0.09.7 ± 0.6
MFM: No visual observationsNone (structured state, action & outcome text)8.7 ± 0.69.7 ± 0.6

Table 4 (revised paper; † = added in revision, including all GPT-6-Astra results). Ablation of MFM's components (milestones; mean ± std over 3 seeds):

VariantSol Diamond (/ 10)Sol Dragon (/ 24)Astra Diamond (/ 10)Astra Dragon (/ 24)
Full MFM9.7 ± 0.610.7 ± 0.610.0 ± 0.016.0 ± 0.0
No visual obs.8.0 ± 1.08.3 ± 0.68.7 ± 0.69.7 ± 0.6
Flat8.7 ± 0.68.7 ± 0.69.3 ± 0.69.3 ± 0.6
No agent procedures8.7 ± 0.69.0 ± 1.08.7 ± 0.610.7 ± 1.2
No consolidation8.3 ± 0.68.3 ± 0.68.7 ± 0.69.3 ± 0.6
No working memory †6.7 ± 1.211.5 ± 3.59.3 ± 0.614.0 ± 1.0
Frozen knowledge †7.7 ± 1.510.0 ± 0.09.0 ± 0.011.5 ± 2.1
No evidence links †8.3 ± 0.69.0 ± 0.08.7 ± 0.69.7 ± 0.6
No spatial memory †7.0 ± 1.79.0 ± 0.08.0 ± 1.712.0 ± 2.9
Captions-only †8.0 ± 1.79.0 ± 0.09.0 ± 0.09.7 ± 0.6

We have updated Sections 3.3 and 4.3 and Appendices B and D.3 to explain why visual details needed later require stored frames and to add caption-only controls.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Overall Response (1/2)

We thank the Area Chair and reviewers for their feedback. Our revision (additions in blue; original text preserved) adds, as suggested by the reviewers, 17 controlled baselines and ablations (Tables 4, 7), a 106-query retrieval benchmark (Fig. 16), and 10-life scaling and transfer studies (Figs. 18–19), grouped by reviewer question:

1. Raw Visual Observations vs. Textual Descriptions, and Additional Prior Methods — Reviewer JhJR Q1; Reviewer CSfu W1, Q1; Reviewer pCGr W1, W2, Q1

We updated the Abstract, Sections 1–2, 3.3, and 4.1–4.3, and Appendices B and D.3 to add the MemGPT [1], Generative Agents [2], and Cradle [6] baselines and the caption-only and spatial-memory ablations.

Keeping raw visual observations, and finding them through searchable text cues, lets the agent recover details whose importance only becomes clear later. It outperforms richer text descriptions and six additional prior memory systems. Prior systems convert observations to text when they are recorded (MemGPT, Generative Agents), retrieve keyframes or places without editable notes (JARVIS-1 [3], Optimus-1 [4], MrSteve [5], Cradle), or store code skills with no link to the episodes they came from (Voyager [7]). MFM keeps every episode unchanged and links the agent's editable files to those episodes through a searchable index, so the context loaded at each model call depends on what the agent retrieves, not on how many episodes are stored. With GPT-6-Astra, storing 60–150 word captions instead of images lowers MFM from 10.0 to 9.0 milestones on Mine Diamond and from 16.0 to 9.7 on Ender Dragon (Table 7), about the same as storing no images (8.7 and 9.7, Table 4). In KiCad, removing past images or consolidating them lowers mean reward from 0.440 to 0.427. Captions leave out details that later matter, and they cannot be cropped or compared side by side with the current view. With GPT-5.6-Sol on Mine Diamond, every single-component ablation (6.7–8.7) stops before mining a diamond and falls below VISTA [8] (9.3), which keeps 4 MB of recent frames in context and calls the model 3–4× as often as MFM. With the same task brief and actions, MFM (16.0 ± 0.0) outperforms VISTA [8] (12.7 ± 0.6), PRO-LONG [9] (11.0 ± 1.0), JARVIS-1/Optimus-1 [3, 4] (10.0 ± 4.0), MemGPT [1] (10.0 ± 0.0), Generative Agents [2] (9.0 ± 1.0), MrSteve [5] (7.3 ± 1.5), and Voyager [7] (6.0 ± 1.7) on Ender Dragon (Table 7), and Cradle [6] on KiCad (mean reward 0.373 ± 0.009 vs. 0.440 ± 0.013, Table 3).

2. Self-Evolving Consolidation, Reusable Knowledge vs. Working Memory, and Procedure Creation — Reviewer JhJR W2; Reviewer CSfu W2, W4, Q3; Reviewer cnUW W1, W2; Reviewer pCGr W1, Q1–Q3

We updated Sections 2, 3.4, 4.1, and 4.3 and Appendices A.4.3, B, and D.3 to describe episode boundaries, when memory is consolidated, and memory retention across deaths, and to add the working-memory, frozen-knowledge, evidence-link, and procedure-store experiments and the memory audit.

MFM's gains come from updating notes and procedures that are linked to stored episodes, not from a working scratchpad or file-writing permissions alone. An audit of 5 Minecraft runs (Appendix A.4.3) found all 46 checkable statements in the agent's files (coordinates, game rules, or what a procedure does) matched the run logs, and all 5 cases where stored memory contradicted a later observation (e.g., a pond recorded to the east that lay north) were corrected, 3 within the run and 2 in the next life. Reflexion [10] appends text after each trial and A-MEM [11] updates text memories (cf. MineEvolve [12]); MFM instead links notes and procedures to the visual episodes they came from. The agent updates these files on four events: a predicted outcome is confirmed, a rule or coordinate is verified, a safety reflex interrupts an action, or the agent takes damage or dies. When a run ends, the harness also indexes it for the next run. With GPT-6-Astra, deleting the working-memory file before every model call still reaches 9.3 ± 0.6 milestones on Mine Diamond and 14.0 ± 1.0 on Ender Dragon (1.3 above VISTA [8]). Freezing notes and procedures while keeping working memory lowers these to 9.0 ± 0.0 and 11.5 ± 2.1, and removing the episode links lowers them to 8.7 ± 0.6 and 9.7 ± 0.6 (Tables 4, 7). All methods can write files and run Python and keep their memory across deaths (Table 6), and PRO-LONG [9] already logs a structured record of each action (Fig. 2). Giving PRO-LONG and VISTA [8] MFM's procedure folder and consolidation prompt yields 13.0 ± 1.7 and 12.3 ± 3.2 on Ender Dragon (Table 7): PRO-LONG's logs contain no images to check terrain (e.g., headroom over lava), and VISTA's stored frames have no recorded outcomes to debug scripts against. Across 3 Ender Dragon runs, MFM wrote 12 scripts, revised 6, and retired 1 inherited script.

AuthorsOpenReview ↗

Overall Response (2/2)

3. Cue-Based Retrieval (Retrieval by Contextual Predicates) vs. Multimodal Embeddings and Effect on Navigation — Reviewer JhJR Q2; Reviewer CSfu W3, Q2, Q3; Reviewer cnUW W3

We updated the Abstract, Sections 1, 4.4, and 6, and Appendices A.4.1–A.4.2 and D.3 to add the retrieval benchmark and its miss breakdown, MFM with embedding retrieval (MFM + Embed), an analysis of how retrieved frames change the agent's plans, and failure cases on visual queries.

Searching structured cues and then checking the retrieved frames keeps top-1 retrieval accuracy at 93.4% when other runs' episodes are mixed in, vs. 2.8%–44.3% for the two embedding baselines. To measure relevance directly rather than the age of retrieved episodes (Fig. 6), we built 106 ground-truth queries over 5 Minecraft runs (37 about events, 40 about places, 20 about times, and 9 about visual content) and added episodes from three other runs as distractors; returning one counts as a miss. MFM answers 99 of 106 correctly, including all 97 event, place, and time queries but only 2 of 9 visual queries, and never returns a distractor episode. The embedding + LLM reader baseline (GPT-6-Astra reads the top-20 hits and views the top-8 frames) answers 44.3%, and the embedding-only baseline 2.8% (Fig. 16). Replacing MFM's search with bge-small text and CLIP image embedding retrieval, all else identical, gives 14.7 ± 1.2 milestones on Ender Dragon, vs. 16.0 ± 0.0 for MFM and 10.0 ± 4.0 for JARVIS-1/Optimus-1, which uses the same embeddings without MFM's memory structure (Table 7). MFM queried memory in 14% of 1,694 model calls, and the next plan used the returned coordinates 91% of the time after a location query and 86% after a time-range query. After viewing a frame, the agent added a new navigation target 2.5× as often on Mine Diamond (17% vs. 7%), and 2.4× as often on Ender Dragon (33% vs. 14%), and 88% (57/65) of those targets were within 8 blocks of the retrieved episode (Fig. 17).

4. Multi-Run Memory Scalability (0–10 Lives) and Sources of Cross-Task Transfer — Reviewer JhJR W1; Reviewer CSfu W3, W4; Reviewer cnUW W4

We updated Sections 1, 4.2, 5.1–5.2, and 6 and Appendix D.4 to add the inherited world-memory experiment, results for up to 10 stored lives, the transfer comparisons, and storage costs.

MFM's memory keeps working as experience accumulates. We stored 0 to 10 earlier lives, each a full 36,000-step Ender Dragon run (4,806 to 47,035 episodes; 0.23 to 2.06 GB). Median query time rises from 399 to 899 ms, and top-1 accuracy on 46 ground-truth queries stays at 87.0%–91.3% (vs. 15% and 0% for the two embedding baselines at 10 lives, Fig. 18). Within a single life, new input tokens per model call stay nearly constant (2.6k to 2.9k). Separate experiments distinguish three ways stored experience transfers. First, agent-written scripts alone, copied into a new world with no episodes or notes, cut the steps needed for three new tasks by 30%, 7%, and 26% (Table 5). Second, in KiCad, where the agent writes no scripts, reusing past GUI action sequences and checklists raises mean reward from 0.413 to 0.440 with GPT-6-Astra (Table 3, Figs. 14–15). Third, memory inherited from earlier exploration of the same world raises ore-mining completion from 8.3% (1/12) with none inherited to 33.3% (2/6), 62.5% (5/8), and 91.7% (11/12) as more is inherited (Section 5.2, Fig. 19). The costs are storage and search time: with 42k episodes from earlier lives, the slowest location query takes 3.4 s and lossless images take up about 94% of the archive, which motivates database memory and removal of near-duplicate frames (Section 6, Appendix D.4).

References

  • [1] Packer et al. MemGPT: Towards LLMs as Operating Systems. arXiv, 2023.
  • [2] Park et al. Generative Agents: Interactive Simulacra of Human Behavior. UIST, 2023.
  • [3] Wang et al. JARVIS-1: Open-World Multi-Task Agents with Memory-Augmented Multimodal Language Models. TPAMI, 2025.
  • [4] Li et al. Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks. NeurIPS, 2024.
  • [5] Park et al. MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory. ICLR, 2025.
  • [6] Tan et al. Cradle: Empowering Foundation Agents Towards General Computer Control. arXiv, 2024.
  • [7] Wang et al. Voyager: An Open-Ended Embodied Agent with Large Language Models. TMLR, 2024.
  • [8] Han et al. VISTA: A Visual Harness for Reasoning in an Interactive World. arXiv, 2026.
  • [9] Fox et al. PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning. arXiv, 2026.
  • [10] Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS, 2023.
  • [11] Xu et al. A-MEM: Agentic Memory for LLM Agents. NeurIPS, 2025.
  • [12] Xie et al. MineEvolve: Self-Evolution with Accumulated Knowledge for Long-Horizon Embodied Minecraft Agents. arXiv, 2026.
Area_Chair_jQuSOpenReview ↗

Please review the rebuttal and provide further comments as appropriate

Dear Reviewers,

The authors have submitted their rebuttal. Please review it and provide further comments as appropriate.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.