- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes MFM, an external memory for long-horizon multimodal agents laid out as a filesystem. The harness writes immutable records of each procedure-call "episode" (frames, state, action, outcome). It also writes a JSONL index of cues (steps, x/y/z, action kind, tags, outcome summary) that point back to those records, plus a grid of traversed cells. The agent writes revisable notes, a guide, executable procedures, and a working plan, each citing its supporting episodes. Retrieval first searches the cues and then inspects the referenced frames (crop, contact sheet, then-vs-now comparison), and a periodic consolidation step rewrites the agent's memory. The authors map these pieces onto episodic, spatial, semantic, procedural, and working memory. Experiments cover two MCU Minecraft tasks and 25 CUABench-KiCad tasks with three frontier models, against a no-memory agent and reimplementations of PRO-LONG and VISTA. With GPT-6-Astra, MFM reaches 10/10 Diamond milestones and 16/24 Dragon milestones (strongest baseline: 12.7), and 0.440 vs. 0.400 KiCad reward.
Critical strengths
- The experimental comparison is controlled and valid. All methods share the same 28 closed-loop atoms, 12 KB brief, plan cap, and [RECENT]/[MAP] block. The comparison spans three models, two Minecraft tasks, and a GUI domain. On Ender Dragon with GPT-6-Astra, MFM reaches 16.0±0.0 milestones vs. 12.7±0.6 for VISTA, and Fig. 7 shows the baselines plateauing. In the one-run cost comparison (Table 8), MFM is also the cheapest.
Critical weaknesses
-
The ablation does not isolate components. Table 4 uses only GPT-5.6-Sol on Minecraft, the setting with the smallest margin. Every single-component ablation falls below VISTA (Diamond 8.0–8.7 vs. 9.3). That includes "No agent procedures" and "No consolidation", which still keep the index and matched inspection. Spatial and working memory, two of the five headline memory forms, are never ablated. Ablations on GPT-6-Astra/Dragon and on KiCad are needed.
-
The novelty is limited; there are missing baselines and related work. The ingredients are established:
- retrieval over a memory stream plus reflection (Generative Agents, Park et al., 2023);
- agent-managed tiered memory (MemGPT, Packer et al., 2023);
- skill libraries (Voyager);
- multimodal experience pools (JARVIS-1, Optimus-1);
- place-event memory (MrSteve);
- screenshot-based episodic memory for computer control (Cradle, Tan et al., 2024).
Questions for authors
-
Could the authors report the ablations with GPT-6-Astra on Dragon? Why does every GPT-5.6-Sol ablation fall below VISTA?
-
Do the reimplemented PRO-LONG and VISTA keep their memory across deaths ("lives")?
-
For each model, what scaffold, context policy, and reasoning settings were used? What triggers consolidation, and how are episode boundaries defined?
6 comments
Response to Reviewer pCGr (2/3)
[W2] Controlled Comparison Against Six Prior Memory Architectures. We thank the reviewer and agree that each listed ingredient has precedent and that MFM does not introduce them individually; Sec. 2 and Table 1 now include Generative Agents and MemGPT. Our contribution is an embodied-agent memory architecture built on one design decision: raw multimodal episodes are kept unchanged, and everything the agent writes (notes, guides, procedures, plans) cites the episodes it came from. The agent can thus revisit original frames when a visual detail matters later, and check the evidence behind a note or script before revising it. Prior systems depart from this differently: Generative Agents and MemGPT convert experience to text at write time; JARVIS-1, Optimus-1, MrSteve, and Cradle keep frames or places, but nothing the agent writes links back to them; Voyager's skills are detached from their source episodes. As suggested, we ran adapted versions under the same brief, action atoms, and GPT-6-Astra settings (3 seeds; Table 7). On Ender Dragon they reach 6.0–10.0 milestones vs. 16.0 for MFM and 12.7 for VISTA; on KiCad, Cradle reaches 0.373 vs. 0.440. These are our adaptations, not the original full pipelines (Appendix D.3).
| Architecture (GPT-6-Astra, 3 Seeds; Dragon / 24 unless noted) | Benchmark Score | Memory Mechanism & Observed Bottleneck |
|---|---|---|
| Full MFM (Ours) | 16.0 ± 0.0 | Cue index + raw PNGs + grid.jsonl + linked mine/ |
| MFM + Multimodal Embed | 14.7 ± 1.2 | Predicate search → bge-small-en-v1.5 + CLIP ViT-B/32 rank fusion; all else identical |
| JARVIS-1 / Optimus-1 [4, 5] | 10.0 ± 4.0 | Plan keyframes + outcomes (bge + CLIP); no editable revision |
| MemGPT [6] | 10.0 ± 0.0 | Core memory + FIFO text summaries + bge |
| Generative Agents [7] | 9.0 ± 1.0 | Recency + importance + bge + reflections; no frames |
| MrSteve [8] | 7.3 ± 1.5 | CLIP place memory; no outcome cues or notes |
| Voyager [9] | 6.0 ± 1.7 | Skill library |
| Cradle [3] (KiCad, 25 tasks) | 0.373 ± 0.009 | Rolling screenshot buffer; trails MFM (0.440), MFM w/o cross-task (0.413), VISTA [1] (0.400) |
Table 7 (new in revision). Additional baselines and controls (milestones; Diamond / 10, Dragon / 24; same brief and action atoms; — = not run):
| Method / Control Variant | Sol Dia. | Sol Drag. | Astra Dia. | Astra Drag. |
|---|---|---|---|---|
| Main-table reference methods (Table 2) | ||||
| MFM (ours) | 9.7 ± 0.6 | 10.7 ± 0.6 | 10.0 ± 0.0 | 16.0 ± 0.0 |
| VISTA | 9.3 ± 0.6 | 9.0 ± 1.0 | 9.7 ± 0.6 | 12.7 ± 0.6 |
| PRO-LONG | 8.0 ± 1.0 | 8.3 ± 0.6 | 9.3 ± 0.6 | 11.0 ± 1.0 |
| Direct Evaluation | 4.0 ± 1.0 | 6.0 ± 1.0 | 6.3 ± 0.6 | 6.3 ± 0.6 |
| Adapted prior-paradigm & multimodal-embedding baselines | ||||
| MFM + Multimodal Embed | — | — | — | 14.7 ± 1.2 |
| JARVIS-1 / Optimus-1 | — | — | — | 10.0 ± 4.0 |
| MemGPT | — | — | — | 10.0 ± 0.0 |
| Generative Agents | — | — | — | 9.0 ± 1.0 |
| MrSteve | — | — | — | 7.3 ± 1.5 |
| Voyager | — | — | — | 6.0 ± 1.7 |
| Representation, consolidation & procedure-store controls | ||||
| MFM w/o working memory | 6.7 ± 1.2 | 11.5 ± 3.5 | 9.3 ± 0.6 | 14.0 ± 1.0 |
PRO-LONG + MFM procedures/ | — | — | — | 13.0 ± 1.7 |
VISTA + MFM procedures/ | — | — | — | 12.3 ± 3.2 |
| MFM w/o spatial memory | 7.0 ± 1.7 | 9.0 ± 0.0 | 8.0 ± 1.7 | 12.0 ± 2.9 |
| MFM w/ frozen knowledge | 7.7 ± 1.5 | 10.0 ± 0.0 | 9.0 ± 0.0 | 11.5 ± 2.1 |
| MFM w/o evidence links | 8.3 ± 0.6 | 9.0 ± 0.0 | 8.7 ± 0.6 | 9.7 ± 0.6 |
| MFM w/ write-time captions | 8.0 ± 1.7 | 9.0 ± 0.0 | 9.0 ± 0.0 | 9.7 ± 0.6 |
Table 1 (revised; † = new row). MFM vs. prior agent memory systems:
| Method | Raw multimodal evidence | Index with evidence pointers | Spatial memory | Agent-writable memory | Knowledge linked to evidence | Retrieval |
|---|---|---|---|---|---|---|
| Generative Agents † | ✗ | ✗ | ✓ | ✓ | ✓ | Recency + importance + embedding relevance |
| MemGPT † | ✗ | ✗ | ✗ | ✓ | ✗ | Text, timestamp, and embedding search |
| Voyager | ✗ | ✗ | ✗ | ✓ᵃ | ✗ | Embedding similarity |
| JARVIS-1 | ✓ᵇ | ✗ | ✗ | ✓ᵇ | ✗ | Multimodal similarity |
| MrSteve | ✓ | ✗ | ✓ | ✗ | ✗ | Place/event lookup |
| PRO-LONG | ✗ᶜ | ✗ | ✗ | ✓ᵈ | ✗ᵈ | Regex / code over log |
| VISTA | ✓ | ✗ᵉ | ✗ | ✓ᵉ | ✗ | Inspect by frame ID |
| MFM (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | Cue query + evidence inspection |
ᵃ Skill library (code) only. ᵇ Successful trajectories only. ᶜ Text log only. ᵈ Workspace files/notes; structured text log lacks visual frames/links (revised). ᵉ Frame IDs without a searchable index; free-form notes.
We updated the Abstract, Sections 1–2 and 4.2, and Appendix D.3 with these comparisons.
Response to Reviewer pCGr (3/3)
[Q2] Do the reimplemented PRO-LONG and VISTA keep their memory across deaths? We thank the reviewer for these questions. Yes. In our reimplementations, PRO-LONG, VISTA, and MFM all keep 100% of their persistent memory across deaths within a run. PRO-LONG keeps its full logs.txt and workspace; VISTA keeps WORKING.md, GUIDE.md, and its stored frames; and MFM keeps its episodes, index, and agent-written files. The same holds across the 25 sequential KiCad tasks. The main Minecraft benchmarks are single runs that start with no prior memory, so retention there applies to deaths within the run. We now state this in Sec. 4.1 and Table 6.
[Q3] Scaffolds, context policy, reasoning settings, consolidation triggers, and episode boundaries. We added these details to Sections 3.4 and 4.1 and Appendix D.3. All methods use high reasoning effort with every model. The only exceptions are internal helper calls inside the Generative Agents and MemGPT baselines (importance scoring and summarization), which use low effort. MFM and PRO-LONG run in persistent Codex CLI (v0.153.4, bwrap sandbox) or Claude Code CLI sessions, with automatic context compaction at 100k tokens. VISTA runs in its tool-server loop or in Claude Code, with its own checkpoint once stored images reach 4 MB. The Direct baseline makes stateless single API calls. At each call, MFM, PRO-LONG, and VISTA all receive the current frame plus state, recent-plan, and map blocks. In addition, MFM receives its working-memory file, PRO-LONG the tail of logs.txt, and VISTA its WORKING and GUIDE files and inspect tool. Direct receives only the frame and state. In Minecraft, an episode is one plan entry: one primitive action with its repeat count, or one procedure call. The harness saves the frame and state before the action at its first step, and saves the action, outcome, and index entry at its last step. In KiCad, an episode is one batch of GUI actions. Consolidation works as described in our reply to Reviewer JhJR. The agent maintains working.md continuously across compactions. It writes to durable files only on four events: a predicted outcome is confirmed, a rule or coordinate is verified, a safety reflex interrupts an action, or the agent takes damage or dies. At run boundaries, the harness indexes each finished run under previous/.
| Method | Scaffold & Session Policy | Context Given at Each Model Call | Across Deaths |
|---|---|---|---|
| MFM | Codex CLI v0.153.4 (bwrap) / Claude Code CLI; persistent, 100k auto-compact | Frame + [STATE] + [RECENT] + [MAP] + [WORKING] | Keeps episodes/, index.jsonl, mine/ |
| PRO-LONG [2] | Codex CLI v0.153.4 (bwrap) / Claude Code CLI; persistent, 100k auto-compact | Frame + [STATE] + [RECENT] + [MAP] + logs.txt tail | Keeps full logs.txt & workspace |
| VISTA [1] | Tool-server loop / Claude Code CLI; persistent, 4 MB image auto-checkpoint | Frame + [STATE] + [RECENT] + [MAP] + WORKING/GUIDE + inspect | Keeps WORKING.md, GUIDE.md, frames |
| Direct | Stateless single API calls (GPT-6-Astra / GPT-5.6-Sol / Claude Opus 5); no persistent session | Current frame + [STATE] only (no [RECENT], [MAP], or files) | None (memory-free baseline) |
Table 6 (revised paper). Configuration of each method on Minecraft tasks:
| Capability | Direct | PRO-LONG | VISTA | MFM |
|---|---|---|---|---|
| Standing brief | 12 KB | 12 KB | 12 KB | 12 KB |
[RECENT] / [MAP] in observation block | No | Yes | Yes | Yes |
| Episode store and query tool | No | Structured per-action log (logs.txt) via regex/code; no visual episode store (revised) | Archived frames via inspect | Yes |
| Write access to own files; procedure execution | Yes | Yes | Yes | Yes |
| Agent-written procedures (observed) | 0 | 0 | 0 | 4 written / 14 calls (Diamond; 12 across 3 Dragon seeds) |
| Plan cap | 400 steps | 400 steps | 400 steps | 400 steps |
| Images per call | Current frame | Current frame | Current + inspected frames | Current + inspected frames |
We have updated Sections 3.4 and 4.1 and Appendices B and D.3 to document the scaffolds, reasoning settings, context compaction, the context given at each model call, memory retention across deaths, episode boundaries, and when memory is consolidated.
Response to Reviewer pCGr (1/3)
[W1 & Q1] Ablations with GPT-6-Astra, on KiCad, and of Spatial and Working Memory, and Why MFM GPT-5.6-Sol Ablations Fall Below VISTA (GPT-5.6-Sol). We thank the reviewer for these questions. As suggested by the reviewer, we extended the ablations along each listed axis (Tables 4 and 7). All ablations now run on both GPT-5.6-Sol and GPT-6-Astra and on both Mine Diamond and Ender Dragon. We added variants that remove spatial memory (no map, grid, or radius queries) and working memory (
working.mddeleted before every call), as well as frozen-knowledge, no-evidence-link, and caption-only variants. With GPT-6-Astra on Ender Dragon, where full MFM reaches 16.0, the ablations range from 9.3 (no consolidation, or no index) to 14.0 (no working memory), and removing spatial memory gives 12.0 ± 2.9. On KiCad with GPT-6-Astra, removing consolidation or history images lowers reward from 0.440 to 0.427, and removing cross-task memory lowers it to 0.413. As for why every GPT-5.6-Sol Diamond ablation falls below VISTA: full MFM leads VISTA there by only 0.4 milestones (9.7 vs. 9.3). The last milestone, mining a diamond, requires descending about 55 blocks and branch-mining within 12,000 steps. MFM shows the model only the current frame and retrieves past frames on demand. VISTA keeps a 4 MB buffer of recent frames in context and calls the model 3–4× as often. Each MFM component makes up for part of that difference, so removing any one of them leaves the agent stalled before reaching diamond. On the longer Ender Dragon task with GPT-5.6-Sol, the picture changes: four ablations match VISTA (9.0) and two exceed it. One of those results runs against our expectations, and we report it as such: deleting working memory with GPT-5.6-Sol on Ender Dragon gives 11.5 ± 3.5, above full MFM's 10.7 ± 0.6, although with high variance.working.mdWORKING/GUIDE[MAP],grid.jsonl,--nearworking.md; freezes notes/scriptsprocedures/index.jsonlmine/writesep_NNNNNNlinks frommine/logs.txtwithout framesWe have updated Sec. 4.3 and Appendix D.3 to add further GPT-5.6-Sol, GPT-6-Astra, and KiCad ablations and to explain why removing any single component drops MFM below VISTA on Mine Diamond.
References