acceptodds
Under review as a conference paper at ICLR 2027

Think Before You Experiment: Efficient Experiment Selection for AI Research Agents

Abstract

Autonomous AI research agents improve solutions through repeated experimentation, but experiments are expensive while LLM inference is comparatively cheap. This asymmetry raises a simple question: which ideas are worth an experiment? Existing agents typically generate and execute candidate ideas sequentially, devoting little computation to deciding which candidate should be tested next. We introduce IdeaMAP, a lightweight plug-in that spends additional inference on experiment selection. IdeaMAP formulates selection as quality-diversity optimization over a semantic space of research ideas, maintaining a MAP-Elites-style archive of executed experiments and prioritizing candidates predicted either to improve existing elites or to fill promising unexplored niches. Across FML-bench-Lite, IdeaMAP improves multiple host research agents and yields agents that outperform existing systems including The AI Scientist v1 and v2, AIDE, and OpenEvolve. IdeaMAP reaches the host agent’s final validation performance using 31% fewer validation experiments and, at the same validation budget, achieves 6% higher final validation performance. These results show that allocating additional inference to experiment selection can improve the efficiency and performance of autonomous research agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
autonomous AI research agents, quality-diversity optimization, MAP-Elites, idea selection, experiment budget efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MmBwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies which candidate idea an AI research agent should spend its next experiment on. The proposed plug-in, IdeaMAP, has the host generate K=2 ideas per step, uses the LLM to predict each idea’s quality, maps ideas to cells of a semantic MAP-Elites archive (embedding → UMAP → 625-cell grid), and executes the one with the largest predicted archive improvement. It is evaluated on FML-bench-Lite with two host agents, plus seven BioML-bench tasks and ablations on PyCIL. The main claim is that IdeaMAP reaches the Autoresearch host’s final validation performance with 31% fewer experiments and gives 6% higher performance at equal budget.

Critical strengths
  1. The experimental study is well structured and transparently reported, with matched selection-rule ablations, a step-level analysis of the acquisition signal, and full per-task results including negative cases and token cost.
  2. To my knowledge, using a quality-diversity archive over idea semantics as a pre-execution selection rule is new for autonomous research agents, whereas prior systems such as OpenEvolve use such archives only to maintain executed programs.
  3. The method changes only the selection step, so it can be attached to existing agents at low cost.
Critical weaknesses
  1. The host-relative gains in Table 1 are within the reported standard deviations, and by Table 6 IdeaMAP beats its host on only 4 of 8 tasks (Autoresearch) and 5 of 8 (AdaptiveSearch), with nearly all of the average gain coming from DomainBed-CM.
  2. Evaluation coverage is limited: 8 tasks × 3 runs gives only 24 trajectories, and the results on the 7 BioML-bench tasks are inconsistent across hosts. Evaluation on additional benchmarks covering more task types and execution environments is needed.
  3. In the cold-start phase the LLM must predict quality with little or no evaluated history, so its estimates may be ungrounded or hallucinated, and the paper does not examine how early mis-selections affect the search trajectory.
  4. The paper never shows the ideas or case studies inside any cell, so readers cannot judge whether the embedding, UMAP and grid pipeline groups ideas into meaningful research directions.
Questions for authors
  1. How accurate and calibrated is q^\hat q in the first steps of the search compared with later steps?
  2. Can you show the actual ideas in several occupied cells and explain what they share and how neighbouring cells differ?
3 comments
AuthorsOpenReview ↗

Response to Reviewer MmBw [1/2]

We thank the reviewer for the careful review and the constructive comments.

We aligned the experiment history that IdeaMAP(AR) shows the LLM with Autoresearch's. In the submitted runs, this history described each experiment at greater length and dropped the oldest ones once it grew long, and the quality predictor received the same shortened history; now the predictor sees all evaluated ideas. IdeaMAP(AR) thus differs from its host only by the plug-in. We reran every IdeaMAP(AR) result in the paper this way, tried no other configuration, and report all of these reruns; the runs of IdeaMAP(AS) and the other agents are unchanged. All numbers below use the reruns; the headline results of IdeaMAP(AR) on FML-bench-Lite are:

IdeaMAP(AR) on FML-bench-LiteSubmittedRerun
Mean NI, Mean-of-3 (gain over host)0.1720 (+0.0122)0.1918 (+0.032)
Mean NI, Best-of-3 (gain over host)0.2132 (+0.0297)0.2430 (+0.0595)
Tasks better than host4 of 85 of 8
Fewer experiments to reach host's final level (Fig. 1b)31%46%
Gain over host at step 100 (Fig. 1b)+6%+12%

Weakness 1. IdeaMAP(AR) is ahead of its host on 5 of 8 tasks (4 in the submission). To account for run-to-run variation, we report the probability of improvement (PoI): the probability that a random IdeaMAP(AR) run beats a random host run on the same task, averaged over tasks. PoI is 0.61 [0.47, 0.74], or 0.57 [0.41, 0.71] without DomainBed-CM (95% CIs bootstrap the runs within each task). An IdeaMAP(AR) run thus beats a host run more often than it loses, with or without DomainBed-CM.

NI scales differ by more than an order of magnitude across tasks (the nine-agent mean is 0.009 on DomainBed-OH and 0.22 on DomainBed-CM), so the mean NI gain comes mainly from DomainBed-CM. Win rate and mean rank in Table 1 compare agents within each task and do not depend on this scale, so they are the fairer comparison. With the reruns, IdeaMAP(AR) is the best of the nine agents on both (Mean-of-3 win rate 0.615, mean rank 3.56). Its mean NI (0.1918) is also the highest of the nine agents, a gain of +0.032 [−0.002, +0.066] over the host. IdeaMAP(AS) is on par with its host (PoI 0.50).


Weakness 2. The two benchmarks agree for both hosts: on BioML-bench, as on FML-bench-Lite, IdeaMAP is ahead of Autoresearch and on par with AdaptiveSearch.

BioML-bench (leaderboard percentile)Mean Δ [95% CI]PoI [95% CI]Tasks better / tied / worse
IdeaMAP(AR) vs Autoresearch+12.25 [+4.18, +20.15]0.65 [0.54, 0.75]5 / 1 / 1
IdeaMAP(AS) vs AdaptiveSearch−2.80 [−14.51, +9.75]0.43 [0.29, 0.56]3 / 1 / 3

The difference between the hosts matches how much each relies on the plug-in: IdeaMAP chooses the executed experiment in about 85% of the steps under Autoresearch but about 28% under AdaptiveSearch, which calls it only in its second search phase. IdeaMAP(AR)'s mean gain does not depend on a single task: leaving any one out, it stays between +7.35 and +14.91. Resampling tasks as well widens the intervals to [−0.21, +27.75] for IdeaMAP(AR) and [−20.87, +11.81] for IdeaMAP(AS).

The result also holds with a second LLM backbone: with Gemini-3.5-Flash-Lite on four FML-bench-Lite tasks (50 steps, 3 runs), IdeaMAP(AR) is again ahead of Autoresearch (PoI 0.69 [0.51, 0.86]). Further benchmarks are expensive to add, since one run executes 100 experiments and takes a median of 37 hours, and each agent's row in Table 1 is already 24 such runs.

AuthorsOpenReview ↗

Response to Reviewer MmBw [2/2]

Weakness 3 / Question 1. The predicted quality q̂ is unreliable very early, especially in the first ten steps, and becomes steadily more reliable as history accumulates.

StepsSpearman ρ(q̂, q) [95% CI]Bias (q̂ − q)
1–10−0.30 [−0.46, −0.11]+0.21
11–300.15 [0.01, 0.29]+0.01
31–1000.32 [0.22, 0.41]+0.01

The realized quality q of an executed idea is its validation NI over the baseline, and ρ is averaged over the 24 IdeaMAP(AR) runs. Early on, q̂ is over-optimistic; from step 11 on it is nearly unbiased and positively correlated with q, and the correlation keeps rising.

An early mis-ranking only changes which of the two generated candidates is executed, and IdeaMAP(AR)'s task-averaged best-so-far curve is above its host's at every one of the 100 steps.


Weakness 4 / Question 2. Cells group related directions rather than near-duplicates, and neighbouring cells hold related variants. On the implemented code of all IdeaMAP(AR) runs, a blind GPT-5.4 judge that sees only two code diffs rates 21% of same-cell pairs, 15% of adjacent-cell pairs and 1% of distant pairs as the same kind of approach. Ideas closer in the grid more often share an approach in code, so cells are coarse regions of related implementations.

Two examples from IdeaMAP(AR) runs show what a cell shares and how its neighbour differs:

  • PyCIL. One cell reschedules one or two existing regularizers by task stage, e.g., distillation strong early and fading later, or prototype preservation ramped up on later tasks. The adjacent cell holds the joint version: one schedule that rescales logit distillation, feature distillation and the prototype loss together.
  • DomainBed-CM. One cell collects generic domain-generalization regularizers (leave-one-domain-out consistency, a domain-adversarial term, cross-domain feature mixup), all with q ≈ 0. Another collects class-conditional suppression of the color signal (a class-conditional color discriminator with gradient reversal), with q ≈ 0.60–0.63; its neighbour holds further class-conditional invariance penalties on the color signal.

We hope this addresses your concerns, and we are happy to answer any further questions.

Reviewer_MmBwOpenReview ↗

Replying to authors

Thank you for the detailed rebuttal and the additional analyses. While I appreciate the authors' efforts, my main concerns regarding the evaluation coverage and the strength of the host-relative gains remain unresolved:

  1. No additional benchmark was added. IdeaMAP is presented as a general plug-in, but the evidence rests on 15 tasks, two benchmarks with mixed results across hosts. One or two further benchmarks are needed to show that the gains generalize, e.g., AutoLab [1] or MLE-bench Lite [2].

  2. By the authors' own statistics, the gains on FML-bench-Lite are still not significant: the PoI intervals include 0.5 and the NI gain interval includes 0. IdeaMAP(AS) is on par with its host on both benchmarks, which does not support the claim that IdeaMAP improves multiple host agents.

  3. The case study is helpful, but only 21% of same-cell pairs are judged as the same kind of approach, against 15% for adjacent cells, so the cells do not appear to form coherent niches. The predictor is also negatively correlated with realized quality in the first ten steps, and no warm-up or control was provided.

Therefore, I will maintain my original rating. Thank you again for your responses.

Reference

[1] Xu et al. AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks? arXiv:2606.05080.

[2] Chan et al. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv:2410.07095.

Reviewer_kYd2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces IdeaMAP, a plug-in framework for improving experiment selection in autonomous AI research agents. The key motivation is that executing experiments is often substantially more expensive than LLM inference, making it worthwhile to spend additional inference compute on selecting promising experiments. IdeaMAP generates two candidate ideas at each selection step and uses an LLM to predict their validation performance based on the experiment history. It organizes ideas into a MAP-Elites archive using UMAP-projected embeddings and selects the candidate with the largest predicted improvement over the current elite in its corresponding cell. The framework is evaluated with Autoresearch and AdaptiveSearch on FML-bench-Lite and BioML-bench, reporting improvements in validation performance and experiment efficiency. The paper also includes ablations of the selection strategy, archive configuration, and model backbones.

Critical strengths

The paper is well organized and easy to follow. The paper addresses a timely and practical problem in autonomous AI research: selecting which experiments to execute under a limited experimental budget. The plug-in design is also practical and potentially applicable to different agent frameworks.

Critical weaknesses

Although the results are promising, the improvements are relatively small compared with the observed variation across runs. On FML-bench-Lite, the average normalized improvements over the two host agents are only +0.0033 for AdaptiveSearch and +0.0122 for Autoresearch (Table 1), with three runs per task. The gains also appear heavily concentrated on DomainBed-CM; excluding this task substantially reduces or eliminates the average advantage (Table 6). Furthermore, the headline claim of 31% fewer experiments is based on an averaged learning curve without uncertainty estimates (Fig. 1b). Additional runs and statistical analyses would strengthen confidence that the improvements are consistent and generalizable.


IdeaMAP generates two candidate ideas per selection step, including a second candidate explicitly prompted to be different, whereas the original host agents generate only one. Consequently, the reported improvements may partly arise from increased candidate generation and diversity rather than the proposed MAP-Elites-based selection strategy. Although Table 3 includes relevant controls, they are limited to PyCIL and one host agent, where random selection already captures a substantial fraction of the improvement. Extending these comparisons across the main benchmarks and both host agents would help establish the specific contribution of the selection mechanism.


The central selection objective, maximizing predicted improvement over a cell’s elite, is closely related to established quality-diversity approaches, particularly BOP-Elites. The paper would benefit from a clearer explanation of how its selection strategy differs from these approaches and what the main methodological novelty is beyond applying quality-diversity principles to research-agent experiment selection. Additionally, the archive-based objective is not fully aligned with the evaluation metric, which primarily measures the best solution rather than aggregate archive quality. This distinction deserves further discussion.

Questions for authors
  1. Could the authors provide additional runs or statistical analyses to establish the reliability of the improvements, particularly when DomainBed-CM is excluded? Could uncertainty estimates also support the claimed 31% reduction in experiments?

  2. How does the proposed selection strategy differ from existing quality-diversity methods, particularly BOP-Elites? How is optimizing local archive improvements justified when evaluation focuses on the best solution?

  3. Could the authors clarify how predicted quality is calibrated, how ties are handled, and how realized local archive improvement is computed in Sec. 4.7?

3 comments
AuthorsOpenReview ↗

Response to Reviewer kYd2 [1/2]

We thank the reviewer for the careful review and the constructive comments.

We aligned the experiment history that IdeaMAP(AR) shows the LLM with Autoresearch's. In the submitted runs, this history described each experiment at greater length and dropped the oldest ones once it grew long, and the quality predictor received the same shortened history; now the predictor sees all evaluated ideas. IdeaMAP(AR) thus differs from its host only by the plug-in. We reran every IdeaMAP(AR) result in the paper this way, tried no other configuration, and report all of these reruns; the runs of IdeaMAP(AS) and the other agents are unchanged. All numbers below use the reruns; the headline results of IdeaMAP(AR) on FML-bench-Lite are:

IdeaMAP(AR) on FML-bench-LiteSubmittedRerun
Mean NI, Mean-of-3 (gain over host)0.1720 (+0.0122)0.1918 (+0.032)
Mean NI, Best-of-3 (gain over host)0.2132 (+0.0297)0.2430 (+0.0595)
Tasks better than host4 of 85 of 8
Fewer experiments to reach host's final level (Fig. 1b)31%46%
Gain over host at step 100 (Fig. 1b)+6%+12%

Weakness 1 / Question 1. IdeaMAP(AR) wins more often than it loses against its host (5 of 8 tasks won), with or without DomainBed-CM; its mean NI gain comes mainly from DomainBed-CM.

IdeaMAP(AR) vs Autoresearch8 taskswithout DomainBed-CM
Mean ΔNI [95% CI]+0.032 [−0.002, +0.066] (+0.0122 in the submission)−0.001 [−0.011, +0.009]
PoI [95% CI]0.61 [0.47, 0.74]0.57 [0.41, 0.71]

PoI (probability of improvement) is the probability that a random IdeaMAP run beats a random run of the compared method on the same task, averaged over tasks. The 95% CIs bootstrap the runs within each task; resampling tasks as well widens them (e.g., PoI 0.61 [0.35, 0.85]) without changing these conclusions.

NI scales differ by more than an order of magnitude across tasks (mean over the nine agents: 0.009 on DomainBed-OH, 0.22 on DomainBed-CM), so DomainBed-CM dominates the mean. Within-task statistics do not depend on this scale, and on the two reported in Table 1, win rate and mean rank (recomputed with the reruns), IdeaMAP(AR) ranks first among the nine agents (Mean-of-3 win rate 0.615 vs 0.542 for its host). It also ranks first on median rank, z-score (NI standardized across the agents within each task) and Bradley–Terry score (a strength score fitted to the per-task pairwise wins), under both Mean-of-3 and Best-of-3. IdeaMAP(AS) is on par with its host (PoI 0.50; mean ΔNI +0.003).

Recomputing the Fig. 1b analysis with the reruns, IdeaMAP(AR) reaches the host's step-100 level with 46% fewer experiments (31% in the submission; 95% CI from no saving to 67%), and at step 100 it is 12% above the host (95% CI: −2% to +26%). IdeaMAP(AR) is also ahead of its host on a second benchmark, BioML-bench, by 12.25 leaderboard-percentile points (95% CI: +4.18 to +20.15).


Weakness 2. With candidates generated the same way, IdeaMAP's choice comes out ahead of random, quality-only and diversity-only choice. On the AdaptiveSearch host, because each run takes about 37 hours, we have these selection controls on five tasks (PyCIL, USB, DomainBed-CM, DomainBed-OH, OpenOOD; GPT-5.4, 100 steps, 3 runs). On PyCIL, these are the runs of Table 3. All arms generate two candidates and differ only in which is executed, so the comparison isolates selection:

IdeaMAP(AS) vsPoI [95% CI]Tasks better / tied / worseMean ΔNI [95% CI]
Random choice0.56 [0.33, 0.76]3 / 0 / 2+0.020 [−0.051, +0.088]
Quality only (higher predicted quality q̂)0.53 [0.33, 0.73]4 / 0 / 1+0.013 [−0.030, +0.065]
Diversity only (farther from its nearest elite)0.77 [0.57, 0.93]5 / 0 / 0+0.050 [+0.010, +0.101]

On the Autoresearch host, the control is an LLM selector that reads the experiment history and picks one of two candidates generated the same way, at the same selection cost (Gemini-3.5-Flash-Lite, 4 tasks, 50 steps, 3 runs). IdeaMAP(AR) is on par with the LLM selector across all pairs of runs (PoI 0.43 [0.19, 0.69]) and leads on Best-of-3 (better on 3 of 4 tasks).

AuthorsOpenReview ↗

Response to Reviewer kYd2 [2/2]

Weakness 3 / Question 2. The selection objective is closely related to that of BOP-Elites, a surrogate-assisted quality-diversity (QD) method, and we do not claim the acquisition as a contribution. What is new is the setting: BOP-Elites optimizes over a parametric search domain with a user-specified descriptor and a Gaussian-process surrogate, and none of these exists when the candidates are text. Instead, IdeaMAP chooses between two ideas generated by the host, computes the descriptor from the idea's text before execution, and uses the host LLM's in-context prediction as the surrogate. Other surrogate-assisted QD methods such as SAIL share BOP-Elites' requirements, and LLM-driven code-search systems (AlphaEvolve, OpenEvolve) archive programs after executing them and do not use the archive to choose which candidate to execute.

Optimizing local archive improvement rarely works against the best-solution metric. A candidate is scored by how much it is predicted to raise the best result in its cell: its full predicted quality q̂ in an empty cell, and only its predicted margin over the elite in an occupied cell. In practice, IdeaMAP(AR) picks the same candidate as choosing by q̂ alone on 75.3% of steps, and where the two choices differ, the final result does not suffer: in the selection controls of Weakness 2, IdeaMAP(AS) ends above quality-only selection on 4 of 5 tasks.

A better archive also goes with a better final result. Across the 48 runs of IdeaMAP(AR) and its host, the Q-D score (the sum of per-cell best results) and the final test NI, both standardized within task, correlate (Pearson 0.53; partly by construction, since a run's best result is one term of its Q-D sum). IdeaMAP(AR) builds the better archive: its run has a higher Q-D score than an Autoresearch run in 63% of the 72 within-task run pairs and covers more cells in 68%.


Question 3. Predicted quality is not calibrated in a separate step: q̂ is the host LLM's in-context prediction from all executed ideas and their realized quality q, clipped to [0, 1]. When the two candidates tie, that is, are predicted to raise the best result of their cells by the same amount, IdeaMAP prefers the candidate in an empty cell, then the one with higher q̂, then the first one. The realized local archive improvement of an executed idea is the amount by which its realized quality q exceeds its cell's elite quality at decision time (taken as 0 for an empty cell), and 0 if q is lower; the predicted improvement is the same with q̂ in place of q.

q̂ becomes more reliable as history accumulates: the within-run Spearman correlation between q̂ and q is −0.30, 0.15 and 0.32 in steps 1–10, 11–30 and 31–100, and the bias of q̂ (its average overestimate of q) falls from +0.21 to +0.01. Ties occur in 3.8% of IdeaMAP(AR) selection steps (9.8% for IdeaMAP(AS)), almost always because neither candidate is predicted to improve its cell.

Predicted and realized improvements correlate within runs (Spearman 0.393 for IdeaMAP(AR) and 0.461 for IdeaMAP(AS), averaged over runs). Because both subtract the same elite quality, part of this correlation would appear even if q̂ carried no information. Shuffling q̂ among the steps of the same run keeps that shared part and removes the information in q̂: the correlation falls to 0.290 (IdeaMAP(AS): 0.189), and no shuffle reaches the observed value. The gap is the information that q̂ adds.


We hope this addresses your concerns, and we are happy to answer any further questions.

Thanks for the thorough response and for rerunning IdeaMAP(AR) with the plug-in as the only difference from its host. Treating experiment selection as a separate decision problem is a timely idea, and as Reviewer MmBw notes, using a quality-diversity archive over idea semantics to select experiments is new for research agents.

My main concern remains whether the archive itself adds value. Based on the reported confidence intervals, the gain over Autoresearch is not statistically significant, and IdeaMAP(AS) performs on par with its host. Given the same two candidates, IdeaMAP’s selection also performs no better than random selection or the LLM selector requested by Reviewer XbZt. This suggests that most of the gain over the host comes from generating a second candidate rather than from the archive. Since the acquisition method is adopted from BOP-Elites, the paper’s contribution depends on showing that the archive helps beyond simpler selection strategies. The current results do not establish this, so I maintain my rating of 2.

Reviewer_XbZtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents IdeaMAP, a plug-in for selecting experiments in automated research agents. It groups research ideas into semantic regions, maintains the best observed result in each region, and selects candidates by their predicted improvement to this archive. Evaluations with two host agents report improved final performance and experiment efficiency, with additional biomedical experiments and component ablations.

Critical strengths

The resource-allocation problem is real and consequential. When experimental execution dominates inference cost, spending more computation on selecting experiments is a sensible systems approach with potential practical value.

The design is modular and evaluated beyond its internal objective. Integration with existing agents and evaluation of final solutions provide useful evidence that the policy can improve an operational research workflow.

Critical weaknesses

The technical novelty is insufficiently established. The method combines established quality-diversity search, semantic embeddings, and LLM quality prediction. The paper does not clearly identify a new technical obstacle that requires this particular combination or explain what substantive insight distinguishes it from a straightforward adaptation of surrogate-assisted quality-diversity optimization.

The necessity of the proposed machinery remains unproven. The quality-only ablation is relevant, but it does not establish an advantage over a capable, history-aware LLM selector explicitly instructed to balance performance, redundancy, and exploration under comparable resource budgets. From a systems perspective, the central question is whether the semantic archive delivers a dependable benefit over simpler selection and memory policies after accounting for its overhead. The current evidence does not resolve this question sufficiently.

Questions for authors

Which specific component or adaptation is technically new relative to existing surrogate-assisted quality-diversity methods? What difficulty specific to research-agent experiment selection does it resolve?

Please compare against the same backbone model given the same candidates and experimental history, explicitly asked to select the next experiment by considering expected performance, redundancy, and exploration. Does IdeaMAP retain an advantage under matched inference and experiment budgets?

Why is maximizing the sum of per-region best results an appropriate proxy for finding the single best final solution? How is continued exploration justified when little experiment budget remains?

How do the authors establish that semantic cells capture redundancy in implemented experiments, rather than similarity in their descriptions? Beyond the existing single-task ablation, please establish whether the grid provides a consistent advantage over simpler similarity-based selection or structured experiment memory.

Under a fixed total resource budget, does IdeaMAP still improve final-solution quality? Please account for selection inference, archive maintenance, and experiment execution, and establish when fewer experiments translate into lower cost or elapsed time.

3 comments
AuthorsOpenReview ↗

Response to Reviewer XbZt [1/2]

We thank the reviewer for the careful review and the constructive comments.

We aligned the experiment history that IdeaMAP(AR) shows the LLM with Autoresearch's. In the submitted runs, this history described each experiment at greater length and dropped the oldest ones once it grew long, and the quality predictor received the same shortened history; now the predictor sees all evaluated ideas. IdeaMAP(AR) thus differs from its host only by the plug-in. We reran every IdeaMAP(AR) result in the paper this way, tried no other configuration, and report all of these reruns; the runs of IdeaMAP(AS) and the other agents are unchanged. All numbers below use the reruns; the headline results of IdeaMAP(AR) on FML-bench-Lite are:

IdeaMAP(AR) on FML-bench-LiteSubmittedRerun
Mean NI, Mean-of-3 (gain over host)0.1720 (+0.0122)0.1918 (+0.032)
Mean NI, Best-of-3 (gain over host)0.2132 (+0.0297)0.2430 (+0.0595)
Tasks better than host4 of 85 of 8
Fewer experiments to reach host's final level (Fig. 1b)31%46%
Gain over host at step 100 (Fig. 1b)+6%+12%

Weakness 1 / Question 1. The new part is making surrogate-assisted quality-diversity (QD) search usable when the candidates, the descriptor and the surrogate's input are all text. The acquisition score is not new: it is the expected improvement over a region's elite used by BOP-Elites, a surrogate-assisted QD method, with the surrogate's uncertainty set to zero.

The difficulty specific to research agents is that an idea is text, which removes the three things surrogate-assisted QD relies on:

  • No search domain. Surrogate-assisted QD methods such as SAIL optimize the acquisition over a parametric space. IdeaMAP chooses between the two ideas the host has just generated, without changing how they are generated.
  • No given descriptor. These methods assign a candidate to a region by features the user specifies. IdeaMAP computes this descriptor from the idea's text before execution and refits the mapping during the run.
  • No trainable surrogate. An idea has no parameterization for a Gaussian process, and a run has at most 100 observations. IdeaMAP's surrogate q̂ is the host LLM's in-context prediction of an idea's quality and needs no training. LLMs have been used as in-context surrogates in evolutionary search before, but to score numeric candidates; here the surrogate scores a research idea before it is run.

LLM-driven code-search systems (AlphaEvolve, OpenEvolve) also keep archives, but use them after execution, to sample parents and store evaluated programs; none of them chooses among candidates by predicted improvement over an archive elite.


Weakness 2 / Question 2. At the same selection cost, IdeaMAP(AR) is on par with a history-aware LLM selector across all pairs of runs and leads on Best-of-3.

Both use Gemini-3.5-Flash-Lite on the Autoresearch host (Opacus, DomainBed-CM, PrivacyMeter, OpenOOD; 50 steps; 3 runs) and receive two candidates generated the same way at every step. The selector sees the full history that IdeaMAP's predictor sees (every executed idea with its metric and quality), is asked to weigh expected performance, redundancy with past experiments and exploration value, and gets no cell information. Its call costs the same as IdeaMAP's prediction call (3.2K tokens, USD 0.0010 per step), and both spend USD 0.77 per run.

PoI (probability of improvement) is the probability that a random IdeaMAP run beats a random run of the compared method on the same task, averaged over tasks. 95% CIs bootstrap the runs within each task.

IdeaMAP(AR) vsPoI [95% CI]Mean ΔNI [95% CI]Best-of-3 NIBest-of-3: tasks better
LLM selector0.43 [0.19, 0.69]−0.001 [−0.033, +0.032]0.229 vs 0.2023 of 4
Autoresearch (one candidate)0.69 [0.51, 0.86]+0.120 [+0.083, +0.161]0.229 vs 0.0853 of 4

Comparing the three agents within each task, which removes the task scale, IdeaMAP(AR) also has the highest Best-of-3 win rate (share of pairwise wins; 0.750, vs 0.500 for the LLM selector and 0.250 for Autoresearch), Bradley–Terry score (a strength score fitted to the same pairwise wins) and z-score (NI standardized across the agents within each task, then averaged).


AuthorsOpenReview ↗

Response to Reviewer XbZt [2/2]

Question 3. In practice, optimizing the archive does not sacrifice the final best result, and late exploration produces most late gains.

The objective scores a candidate by how much it is predicted to raise the best result in its cell; for a candidate in a new cell, that is simply its predicted quality q̂. IdeaMAP(AR) picks the same candidate as choosing by q̂ alone on 75.3% of steps, so it mostly selects by predicted quality. Where the two choices differ, the final result does not suffer: extending the quality-only ablation of Table 3 to five tasks on the AdaptiveSearch host (GPT-5.4, two candidates generated the same way), IdeaMAP(AS) ends above quality-only selection on 4 of 5 tasks.

A better archive also goes with a better final result. Across the 48 runs of IdeaMAP(AR) and its host, the Q-D score (the sum of per-cell best results) and the final test NI, both standardized within task, correlate (Pearson 0.53; partly by construction, since a run's best result is one term of its Q-D sum). IdeaMAP(AR) builds the better archive: its run has a higher Q-D score than an Autoresearch run in 63% of the 72 within-task run pairs and covers more cells in 68%.

With little budget left, exploration is still justified: in the last 20 steps, picks in new cells produced 15 of the 18 improvements of the best-so-far result (15 of 287 such picks, a higher rate than the 3 of 104 in occupied cells). In 8 of the 24 runs, the final best solution is one of these new-cell picks.


Question 4. Cells group implemented code, not only descriptions. A blind GPT-5.4 judge shown only the two code diffs rates two executed ideas as the same approach far more often when they share a cell:

Pairs rated "same approach"Same cellAdjacent cellsDistant cells
IdeaMAP(AR)21%15%1%
IdeaMAP(AS)36%30%5%

The pattern is not explained by timing. Ideas executed a few steps apart often edit the same code, but among pairs executed more than 5 steps apart, same-cell pairs are still rated the same approach far more often than distant pairs. Code-embedding similarity, an independent measure, agrees.

The grid beats simpler similarity-based selection: a rule that always runs the candidate less similar to its closest idea in the archive ends below IdeaMAP(AS) on all 5 tasks (PoI 0.77 [0.57, 0.93]), so avoiding similar experiments alone does not explain the gain. For structured experiment memory, see the LLM selector (Weakness 2 / Question 2).


Question 5. When IdeaMAP(AR) has spent as much on API calls as the host does in its full 100 steps (all LLM calls counted), its best-so-far validation NI is 4.8% higher in relative terms, and it reaches the host's final validation level at 87% of that spend. This already charges IdeaMAP(AR) for its overhead: its 100 steps cost 1.44x the host's API cost (GPT-5.4). Against the LLM selector, total spend is equal (see Weakness 2 / Question 2).

Fewer experiments lower the total cost whenever the experiments saved outweigh IdeaMAP's higher per-step LLM cost, as the cost-matched result above shows; counting each experiment's compute only strengthens this, since every saved experiment also saves its compute. They also mean shorter elapsed time: the time outside the experiment (LLM calls, embeddings and archive maintenance) is 0.71 min per selection step vs 0.68 min for a host step, against a median experiment of 17.5 min.


We hope this addresses your concerns, and we are happy to answer any further questions.

Thank the authors for the rebuttal. I appreciate the efforts on improving the experiments, e.g., including history-aware LLM baseline. However, my concern on the technical contribution and numerical advantage of the paper remain. e.g., the rebuttal result in Q2 does not demonstrate an advantage for IdeaMAP. Across four tasks with three runs each, its average NI difference from the LLM selector is −0.001, with a 95% confidence interval of [−0.033, +0.032]. I therefore maintain my current rating.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.