acceptodds
Under review as a conference paper at ICLR 2027

LabBook: Harnessing Experimental History for Efficient LLM-Driven Discovery

Abstract

Evolutionary approaches to LLM-driven discovery often generate new programs from a small set of selected ancestors. This keeps contexts manageable but can omit useful evidence from other experiments, whereas including the full experimental history produces long, redundant contexts. We introduce a simple, single-agent discovery harness built around LabBook, an agent-maintained memory that serves two complementary roles: guiding retrieval of relevant evidence from a complete experimental log and informing the generation of new solutions. At each iteration, the same agent combines its memory with retrieved evidence and jointly produces the next program and an updated LabBook. This separates complete history retention from selective context construction, without requiring an explicit population or branching search structure. On 49 Frontier-CS problems, LabBook improves the observed quality–cost trade-off over the evaluated evolutionary baselines with two backbones, while remaining competitive across nine additional mathematical, systems, and heuristic-design tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Scientific Discovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_x418OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a single-agent harness for LLM-driven program discovery, built around LabBook, an agent-maintained memory. At each iteration, the agent performs read-only retrieval actions over an append-only experimental log. These return the iteration-indexed score overview, complete programs, or full evaluator feedback. Finally, the agent jointly produces a complete program and a rewritten LabBook. Experiments cover 49 Frontier-CS problems and 9 additional mathematical, systems, and heuristic-design tasks.

Critical strengths
  1. The design is notably simple. It centers on a compact, agent-maintained memory and on-demand retrieval of past evidence, with no explicit population or branching search structure. With this design, it improves the observed quality–cost trade-off over the evaluated evolutionary baselines.

  2. The main results cover two backbones and, beyond Frontier-CS, nine cross-domain discovery tasks. Cost accounting is careful and transparent. Discovery cost is measured in API dollars, and fresh-input, cached-input, and output token counts are reported together with the rate cards.

Critical weaknesses
  1. In Table 2, No Persistent Memory (retrieval only) nearly matches Full LabBook, whereas No Retrieval (memory only) drops to 0.9057. The persistent LabBook state, the paper's central component, therefore contributes only a small aggregate gain. No Retrieval has a slightly higher mean cost than Full LabBook, even though Full LabBook can issue up to five additional retrieval calls per iteration.

  2. LabBook is inherently sequential. Each iteration involves several dependent LLM calls and one evaluation, whereas population-based methods can run proposals and evaluations in parallel. A budget of 50 iterations per task (Table 1) is short for methods designed for hundreds to thousands of parallel evaluations. Sequential greedy refinement is expected to perform well at such short horizons.

Questions for authors
  1. No memory-aware discovery system is included as a baseline. Combining a rewritten natural-language state with retrieval over a raw log is a common pattern in agent research. Could the authors compare against a memory-aware discovery system?

  2. Could you also run a longer-horizon experiment (e.g., several hundred iterations on a subset of tasks)? This would test whether population-based search regains an advantage, and whether LabBook remains accurate as the experimental log grows.

1 comment

Thank you for the response. I have raised my score. Good job.

Reviewer_haUmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how experimental history should be retained and used in LLM-driven program discovery. It proposes LabBook, a single-agent harness that separates complete history storage from selective proposal-context construction. Experiments on 49 Frontier-CS problems and 9 additional mathematical, systems, and heuristic-design tasks show favorable quality–cost trade-offs against three evolutionary baselines with two LLM backbones.

Critical strengths
  1. The paper addresses a practically relevant information-management problem. Selecting only a few parent programs can discard useful experimental evidence, whereas repeatedly including the full history creates expensive and redundant contexts. Separating complete history retention from selective context construction is a useful perspective on the design of discovery harnesses.
  2. The empirical evaluation covers diverse discovery tasks and two backbones. The Frontier-CS results show a favorable quality–cost trade-off, and the nine additional tasks extend the evaluation beyond a single benchmark. The paper also reports token accounting and provider-specific pricing, rather than treating iteration count as a directly comparable computational budget.
Critical weaknesses
  1. The core mechanisms for storing, retrieving, and using historical information have limited methodological novelty. The main design combines a complete chronological log, an LLM-written summary, index-based retrieval, and joint code–memory generation. These are natural applications of established memory and reflection ideas. Treating the history-to-context mapping as a design object is helpful, but does not by itself introduce a substantially new mechanism for memory organization, evidence selection, or experience reuse. The related-work discussion distinguishes the architecture from prior systems, yet the paper does not sufficiently establish what new technical insight makes this particular combination a research contribution beyond a useful engineering simplification.
  2. The evidence does not sufficiently establish why the simple history mechanism is effective. Section 3.2 argues that a compact semantic map and a homogeneous within-run log make simple retrieval sufficient. The ablations support the combined harness, but do not directly validate this explanation or show that the memory consistently preserves and retrieves the evidence needed for subsequent discovery. In particular, the paper lacks systematic measurements of retrieval relevance, reuse of failed or non-best experiments, avoided repeated failures, and the downstream effect of retrieved evidence. The case study retrieves iteration 33, which is also the incumbent best program, so this example alone cannot demonstrate the claimed benefit of preserving evidence beyond successful lineages. Controlled memory or retrieval perturbations, together with trajectory-level diagnostics across tasks, would provide more convincing support for the central mechanism.
  3. The baselines do not adequately isolate the contribution from ordinary memory and reflection. The main comparisons use default configurations of OpenEvolve, AdaEvolve, and EvoX. Although the within-harness ablations are useful, they do not establish whether a conventional discovery agent or an evolutionary baseline augmented with a short experimental summary and the same log access would close the gap. Such comparisons would help distinguish the value of the specific LabBook design from the broader benefit of giving an LLM access to historical feedback.
Questions for authors
  1. Can the authors compare LabBook against a simple reflection-based discovery agent and a memory-augmented evolutionary baseline using the same summary budget, log access, backbone, and cost budget? Which advantage remains attributable to the specific LabBook design?
  2. Can the authors quantify how often retrieval accesses failed or non-best experiments, whether those records influence the next program, and whether removing or replacing the selected evidence changes subsequent improvement? This would directly test the proposed explanation for history reuse.
  3. What diagnostics or safeguards did you employ to mitigate memory drift (e.g., hallucinated lessons) during LabBook rewriting? Does the increased accuracy of information retrieval contribute to performance?
Reviewer_b3kwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LabBook, a single-agent LLM-based discovery framework. LabBook is an agent-maintained notebook that summarizes previously attempted approaches, implementation details, lessons learned, and promising directions, while the complete programs and evaluation results are retained in a separate experimental log. At each iteration, the agent receives the latest evaluated program, its feedback, the notebook, and progress signals. It can then retrieve an overview of previous scores or the exact code and evaluation results of selected experiments before generating a new sample and rewriting the notebook. This removes the need for an explicit population or parent-selection mechanism. The authors compare LabBook with OpenEvolve, AdaEvolve, and EvoX on 49 Frontier-CS problems and nine additional mathematical, systems, and heuristic-design tasks using two LLM backbones. In general, LabBook achieves a better quality–API-cost trade-off on Frontier-CS and remains competitive on the additional tasks, while ablations examine the contributions of persistent memory and selective retrieval.

Critical strengths

-- Simple design with broad evaluation. The paper shows that a relatively simple agent with persistent notes and selective retrieval can compete with more elaborate evolutionary search systems. Evaluation across 58 problems and two backbones supports the takeaways.

Critical weaknesses

-- Limited novelty and insufficient comparisons. The combination of compact notes, persistent history, and selective retrieval is close to established memory architectures like Dynamic Cheatsheet and MemGPT. Although the paper acknowledges prior work, its main comparisons are against evolutionary search methods rather than relevant memory-equipped agents. The component ablations help, but cover only nine tasks, one backbone, and 30 iterations, without reported uncertainty. So, it is unclear whether the gains reflect LabBook’s specific design or the benefits of established memory and retrieval mechanisms.

Questions for authors
  1. Could you compare LabBook with adapted versions of MemGPT or Dynamic Cheatsheet under the same model and cost budget, and extend the memory-retrieval ablations to Frontier-CS with uncertainty across repeated runs?
  2. How robust are the efficiency gains under different caching and pricing assumptions? Could you additionally report wall-clock time and evaluator costs, if any?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.