acceptodds
Under review as a conference paper at ICLR 2027

Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory

Abstract

AI agents increasingly operate over extended time horizons, yet their ability to retain, organize, and recall multimodal experiences remains a critical bottleneck. Building effective lifelong memory requires navigating a vast design space spanning architecture, retrieval strategies, prompt engineering, and data pipelines; this space is too large and interconnected for manual exploration or traditional AutoML to explore effectively. We deploy an autonomous research pipeline to discover Omini-Simplemem, a unified multimodal memory framework for lifelong AI agents. Starting from a naïve baseline (F1 = 0.117 on LoCoMo), the pipeline autonomously executes  50 experiments across two benchmarks, diagnosing failure modes, proposing architectural modifications, and repairing data pipeline bugs, all without human intervention in the inner loop. The resulting system achieves state-of-the-art (SOTA) on both benchmarks, improving F1 by +411% on LoCoMo and +214% on Mem-Gallery relative to the initial configurations. Critically, the most impactful discoveries are not hyperparameter adjustments: bug fixes (+175%), architectural changes (+44%), and prompt engineering (+188% on specific categories) each individually exceed the cumulative contribution of all hyperparameter tuning, demonstrating capabilities fundamentally beyond the reach of traditional AutoML. We provide a taxonomy of six discovery types and identify four properties that make multimodal memory particularly suited for autoresearch, offering guidance for applying autonomous research pipelines to other AI system domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Self-evolving agent, Memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N1gJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper applies an existing autonomous research pipeline to an existing text-only memory framework, with the goal of extending it to multimodal inputs and improving its scores on two memory benchmarks. Over dozens of automated experiments, the pipeline proposes hypotheses, edits code, evaluates, and decides whether to keep or revert each change. The resulting system, which the paper presents as the discovered framework, filters inputs by novelty, stores them as units that separate searchable metadata from raw content, and retrieves with hybrid dense and sparse search, a knowledge graph, and staged expansion under token budgets. The paper reports gains over the initial configuration and over several memory baselines across multiple backbone models, along with an ablation and an efficiency analysis. It also categorizes the changes made by the pipeline into six types, argues that bug fixes, architectural changes, and prompt changes contributed more than hyperparameter tuning and therefore lie beyond traditional AutoML, and lists properties that make multimodal memory suitable for autoresearch.

Critical strengths

S1. The optimization process is documented transparently. The appendix lists per-iteration results including failed and reverted experiments, the category of each change, and the prompts used, so readers can see what the agent changed and how much each change contributed. This gives a concrete and inspectable record of how an agent-driven loop behaves on a multi-component system, which is the part of the work most likely to be reusable, and it supports reproducibility if the code and logs are released.

S2. The case study makes visible how much reported scores on memory benchmarks can depend on output format, prompt placement, and data pipeline details rather than on the memory design itself. The largest gains came from such fixes, and the paper reports them openly. This is useful for interpreting comparisons across memory systems, and the paper is also explicit about the conditions under which the approach applies and about its limitations.

Critical weaknesses

W1. The primary contribution is not clearly identified, and the novelty appears limited. The paper offers a memory framework, a taxonomy of discovery types, and guidance on when autoresearch is suitable, without clearly stating which is the main claim. As a memory system, the design combines patterns that are common in retrieval-augmented memory, including hybrid dense and sparse retrieval, a knowledge graph over extracted entities, staged expansion from summaries to raw content, and a split between searchable metadata and raw storage, and it starts from an existing memory codebase without specifying what is new relative to it. As a study of autoresearch, the pipeline is an existing tool applied without modification, and no comparison is made against human-tuned configurations or conventional AutoML, so the claim that the discoveries lie beyond AutoML rests on the authors' own categorization. The largest reported gain comes from fixing a missing output-format parameter rather than from a design idea. Under either reading, it is difficult to identify what new knowledge the paper adds.

W2. The evaluation does not establish that the reported gains generalize beyond the data used to develop the system. The optimization loop runs on small subsets, such as a single conversation for one benchmark, and the final configuration is then evaluated on the full benchmark that contains those subsets, so no held-out split is used. The pipeline also tunes directly toward the evaluation metric, for example by aligning output formats, matching refusal phrases used for scoring, and stating in the prompt that answers are scored by token-level F1, with per-category settings in the final configuration. It is not clear that the compared systems received comparable tuning, which makes the large margins difficult to attribute to the design itself. The state-of-the-art claim on the multimodal benchmark is also anchored to a previously reported number that was obtained with a different backbone. Finally, the main results report no variance or repeated runs. As a result, the evidence supports the claim that the pipeline can fit a benchmark, but not that the discovered system is better in general.

Questions for authors

Q1. One baseline is attributed to a model card reference, and it is not clear which system and version this refers to or how it was adapted to dialogue question answering. More generally, I am curious how each baseline was configured, including whether it received prompt and output-format tuning comparable to what the pipeline applied to the proposed system. It would be helpful to describe this in the paper.

Q2. The trajectory tables in the appendix describe bug fixes, prompt changes, and parameter adjustments, but they do not show where components such as the atomic unit representation, the knowledge graph, the novelty filter, and the staged expansion were introduced. I wonder which of these were produced by the pipeline and which were specified by the authors in the initial instruction or came from the starting codebase. It would also help to clarify what the initial configuration was and whether the bugs that were fixed were in the original codebase or in code written earlier by the agent.

Q3. The adversarial category appears to be included in the overall score, and its score rises sharply after aligning outputs with the refusal phrases used for scoring. I am curious how the results look when this category is excluded and when a semantic metric is used instead of token-level F1. These results would help me judge whether the gains reflect better memory or better alignment with the scoring procedure.

Reviewer_6KHNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article proposes Omni Simple Mem, which utilizes the automated research framework AutoResearchClaw to design and optimize an AI Agent system that supports long-term multimodal memory through approximately 50 autonomous experiments. This system combines multimodal memory units, knowledge graphs, hybrid retrieval, and progressive retrieval to enhance the ability of agents to store and retrieve historical information. The experiment achieved F1 scores of 0.598 and 0.797 on LoCoMo and Mem Gallery, respectively, which are better than the existing memory systems compared in the paper. The core proposition of the paper is that AI can not only automatically adjust hyperparameters, but also autonomously discover architecture improvements, fix program errors, and optimize the entire system.

Critical strengths
  1. Significant improvements were achieved on two benchmarks and five LLM backbones, demonstrating good model adaptability.

  2. Approximately 50 automated experiments were recorded, covering architecture modifications, error fixes, Prompt optimization, and failure rollback, demonstrating the practical potential of automation research.

  3. Combining hybrid retrieval, knowledge graph, and hierarchical memory mechanisms, and verifying the effectiveness of some key components through ablation experiments.

Critical weaknesses
  1. Lack of direct comparison with manual optimization, random search, or other automatic optimization methods. The significant performance improvement comes from bug fixes and output format adjustments, making it difficult to prove that the self-developed framework itself has unique advantages.

  2. The system repeatedly optimizes the F1 and answer formats for specific benchmarks, and some Prompts are even designed directly for scoring rules. Lack of sufficient independent testing makes it difficult to confirm whether improvements can generalize to other tasks.

  3. The claimed 3.5× throughput improvement compares an 8-worker configuration against sequential baselines. Under single-worker execution, Omni-SimpleMem is actually slower than SimpleMem (1.05 vs. 1.68 queries/sec). Without matched parallelism settings, the claimed efficiency advantage is not convincing.

Questions for authors

Please see above.

Reviewer_3PNqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses an autonomous research pipeline, AutoResearchClaw, to extend the text only SimpleMem codebase into a multimodal memory system for agents. The resulting system is called Omni-SimpleMem. It filters redundant inputs with CLIP, VAD and Jaccard overlap, stores each memory as a Multimodal Atomic Unit with a summary, an embedding and a pointer to raw content, and retrieves memories by a set union of FAISS and BM25 results, a knowledge graph search, and a three level pyramid expansion under a token budget. Across about 50 experiments in about 72 hours, the pipeline raises F1 from 0.117 to 0.598 on LoCoMo and from 0.254 to 0.797 on Mem-Gallery. Table 1 compares the system with six memory baselines on five GPT backbones and reports the best overall F1 on both benchmarks. Table 2 gives a component ablation on LoCoMo, Table 3 reports throughput, and Table 4 groups the discoveries into six types. The paper argues that bug fixes, architecture changes and prompt changes each bring larger gains than hyperparameter tuning, and that this goes beyond traditional AutoML.

Critical strengths
  1. The paper documents a full autonomous optimization run in detail. Table 7, Table 8, Table 9 and Appendix G list every iteration, the change made, the F1 after the change, and the reverted experiments. This level of logging is rare and useful for the community that studies research agents. It is the main reason I see value in the paper, because it shows concretely what kind of changes such a pipeline makes on a real multi component system.
  2. The evaluation covers five backbones and six baselines, and the Mem-Gallery gap in Table 1 is large. With GPT-4o the system reaches 0.797 F1 against 0.535 for SimpleMem. If this gap holds under a clean protocol, the system itself would be a useful multimodal memory baseline.
Critical weaknesses
  1. The optimization appears to be done on the test data. Sec. 3.3 says the pipeline iterates on a small subset and then evaluates on a held out test set. However, Table 9 shows that the LoCoMo trajectory is measured on conversation 26 with 199 QA pairs, and this conversation is part of the full LoCoMo set of 10 conversations used for the final 0.598. LoCoMo has no separate training split. On Mem-Gallery, Phase 4 validates on 14 datasets, Phase 7 runs the final configuration four times on the 20 dataset public subset, and the pipeline stops when F1 reaches the ceiling on that same set. So the reported numbers are selected on the evaluation data, while the baselines in Table 1 receive no such tuning. This is the main reason for my rating, because the SOTA claim and the comparison in Table 1 both depend on it.
  2. A large part of the gain comes from fitting the output to the token F1 metric and from fixing problems of the starting point, not from memory design. The start of 0.117 is far below the 0.432 that SimpleMem itself reaches in Table 1, and Iter 1 only adds the missing response format, which gives 175 percent. Iter 5 repairs timestamps that the pipeline itself had corrupted. Iter 4b and Table 4 list evaluation format alignment with gains up to 63 percent, Cat 5 jumps from 0.447 to 1.000 after matching the refusal phrases of the benchmark, and Appendix H.3.4 adds a prompt that tells the model it is scored by token F1. On Mem-Gallery, Phase 1 also replaces gpt-4.1-nano with gpt-4o. So the 411 percent and 214 percent gains in the abstract measure the distance from a broken setup, and the Table 1 gap may mostly reflect answer formatting that the baselines did not get. The claim that the gains come from the architecture, stated at the end of Sec. 4.3, is not supported.
Questions for authors
  1. Was conversation 26 of LoCoMo excluded from the final LoCoMo score? Please report the final F1 on the nine conversations that the pipeline never saw, and state exactly which Mem-Gallery datasets were used during optimization and which were used for the final score. Appendix B says the 20 dataset subset is used, while Sec. 4.1 describes 1,711 QA pairs from 240 dialogues. Please clarify which one Table 1 uses.
  2. Were the baselines in Table 1 run with the same answer prompt, the same JSON output format, and the same category format constraints of Appendix H? If not, please give baseline results with the same answer generation prompt, so that the comparison isolates the memory design. Please also report an LLM judge accuracy next to token F1.
  3. Several settings conflict. Sec. 3.2.3 gives a default threshold of 0.85 and a budget of 2,000 tokens, while Sec. 4.1 and Table 5 give 0.4 and 6,000. Sec. 4.1 says dense retrieval uses all-MiniLM-L6-v2 with 384 dimensions, while Table 6 lists text-emb-3-large with 3072 dimensions for LoCoMo. Figure 1 reports 0.793 on Mem-Gallery, while the text reports 0.797. Please state which settings produce Table 1.
  4. Table 2 averages over four backbones. Which four, and what is the variance? Please add ablations of the knowledge graph and the novelty filter, and an ablation on Mem-Gallery, since these are the multimodal parts of the system.
  5. What did humans do in the outer loop? Please describe the instructions, the selection of benchmarks and subsets, and any manual restarts or edits during the 72 hours.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.