- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper applies an existing autonomous research pipeline to an existing text-only memory framework, with the goal of extending it to multimodal inputs and improving its scores on two memory benchmarks. Over dozens of automated experiments, the pipeline proposes hypotheses, edits code, evaluates, and decides whether to keep or revert each change. The resulting system, which the paper presents as the discovered framework, filters inputs by novelty, stores them as units that separate searchable metadata from raw content, and retrieves with hybrid dense and sparse search, a knowledge graph, and staged expansion under token budgets. The paper reports gains over the initial configuration and over several memory baselines across multiple backbone models, along with an ablation and an efficiency analysis. It also categorizes the changes made by the pipeline into six types, argues that bug fixes, architectural changes, and prompt changes contributed more than hyperparameter tuning and therefore lie beyond traditional AutoML, and lists properties that make multimodal memory suitable for autoresearch.
Critical strengths
S1. The optimization process is documented transparently. The appendix lists per-iteration results including failed and reverted experiments, the category of each change, and the prompts used, so readers can see what the agent changed and how much each change contributed. This gives a concrete and inspectable record of how an agent-driven loop behaves on a multi-component system, which is the part of the work most likely to be reusable, and it supports reproducibility if the code and logs are released.
S2. The case study makes visible how much reported scores on memory benchmarks can depend on output format, prompt placement, and data pipeline details rather than on the memory design itself. The largest gains came from such fixes, and the paper reports them openly. This is useful for interpreting comparisons across memory systems, and the paper is also explicit about the conditions under which the approach applies and about its limitations.
Critical weaknesses
W1. The primary contribution is not clearly identified, and the novelty appears limited. The paper offers a memory framework, a taxonomy of discovery types, and guidance on when autoresearch is suitable, without clearly stating which is the main claim. As a memory system, the design combines patterns that are common in retrieval-augmented memory, including hybrid dense and sparse retrieval, a knowledge graph over extracted entities, staged expansion from summaries to raw content, and a split between searchable metadata and raw storage, and it starts from an existing memory codebase without specifying what is new relative to it. As a study of autoresearch, the pipeline is an existing tool applied without modification, and no comparison is made against human-tuned configurations or conventional AutoML, so the claim that the discoveries lie beyond AutoML rests on the authors' own categorization. The largest reported gain comes from fixing a missing output-format parameter rather than from a design idea. Under either reading, it is difficult to identify what new knowledge the paper adds.
W2. The evaluation does not establish that the reported gains generalize beyond the data used to develop the system. The optimization loop runs on small subsets, such as a single conversation for one benchmark, and the final configuration is then evaluated on the full benchmark that contains those subsets, so no held-out split is used. The pipeline also tunes directly toward the evaluation metric, for example by aligning output formats, matching refusal phrases used for scoring, and stating in the prompt that answers are scored by token-level F1, with per-category settings in the final configuration. It is not clear that the compared systems received comparable tuning, which makes the large margins difficult to attribute to the design itself. The state-of-the-art claim on the multimodal benchmark is also anchored to a previously reported number that was obtained with a different backbone. Finally, the main results report no variance or repeated runs. As a result, the evidence supports the claim that the pipeline can fit a benchmark, but not that the discovered system is better in general.
Questions for authors
Q1. One baseline is attributed to a model card reference, and it is not clear which system and version this refers to or how it was adapted to dialogue question answering. More generally, I am curious how each baseline was configured, including whether it received prompt and output-format tuning comparable to what the pipeline applied to the proposed system. It would be helpful to describe this in the paper.
Q2. The trajectory tables in the appendix describe bug fixes, prompt changes, and parameter adjustments, but they do not show where components such as the atomic unit representation, the knowledge graph, the novelty filter, and the staged expansion were introduced. I wonder which of these were produced by the pipeline and which were specified by the authors in the initial instruction or came from the starting codebase. It would also help to clarify what the initial configuration was and whether the bugs that were fixed were in the original codebase or in code written earlier by the agent.
Q3. The adversarial category appears to be included in the overall score, and its score rises sharply after aligning outputs with the refusal phrases used for scoring. I am curious how the results look when this category is excluded and when a semantic metric is used instead of token-level F1. These results would help me judge whether the gains reflect better memory or better alignment with the scoring procedure.