- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a single-agent harness for LLM-driven program discovery, built around LabBook, an agent-maintained memory. At each iteration, the agent performs read-only retrieval actions over an append-only experimental log. These return the iteration-indexed score overview, complete programs, or full evaluator feedback. Finally, the agent jointly produces a complete program and a rewritten LabBook. Experiments cover 49 Frontier-CS problems and 9 additional mathematical, systems, and heuristic-design tasks.
Critical strengths
-
The design is notably simple. It centers on a compact, agent-maintained memory and on-demand retrieval of past evidence, with no explicit population or branching search structure. With this design, it improves the observed quality–cost trade-off over the evaluated evolutionary baselines.
-
The main results cover two backbones and, beyond Frontier-CS, nine cross-domain discovery tasks. Cost accounting is careful and transparent. Discovery cost is measured in API dollars, and fresh-input, cached-input, and output token counts are reported together with the rate cards.
Critical weaknesses
-
In Table 2, No Persistent Memory (retrieval only) nearly matches Full LabBook, whereas No Retrieval (memory only) drops to 0.9057. The persistent LabBook state, the paper's central component, therefore contributes only a small aggregate gain. No Retrieval has a slightly higher mean cost than Full LabBook, even though Full LabBook can issue up to five additional retrieval calls per iteration.
-
LabBook is inherently sequential. Each iteration involves several dependent LLM calls and one evaluation, whereas population-based methods can run proposals and evaluations in parallel. A budget of 50 iterations per task (Table 1) is short for methods designed for hundreds to thousands of parallel evaluations. Sequential greedy refinement is expected to perform well at such short horizons.
Questions for authors
-
No memory-aware discovery system is included as a baseline. Combining a rewritten natural-language state with retrieval over a raw log is a common pattern in agent research. Could the authors compare against a memory-aware discovery system?
-
Could you also run a longer-horizon experiment (e.g., several hundred iterations on a subset of tasks)? This would test whether population-based search regains an advantage, and whether LabBook remains accurate as the experimental log grows.
Thank you for the response. I have raised my score. Good job.