- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies which candidate idea an AI research agent should spend its next experiment on. The proposed plug-in, IdeaMAP, has the host generate K=2 ideas per step, uses the LLM to predict each idea’s quality, maps ideas to cells of a semantic MAP-Elites archive (embedding → UMAP → 625-cell grid), and executes the one with the largest predicted archive improvement. It is evaluated on FML-bench-Lite with two host agents, plus seven BioML-bench tasks and ablations on PyCIL. The main claim is that IdeaMAP reaches the Autoresearch host’s final validation performance with 31% fewer experiments and gives 6% higher performance at equal budget.
Critical strengths
- The experimental study is well structured and transparently reported, with matched selection-rule ablations, a step-level analysis of the acquisition signal, and full per-task results including negative cases and token cost.
- To my knowledge, using a quality-diversity archive over idea semantics as a pre-execution selection rule is new for autonomous research agents, whereas prior systems such as OpenEvolve use such archives only to maintain executed programs.
- The method changes only the selection step, so it can be attached to existing agents at low cost.
Critical weaknesses
- The host-relative gains in Table 1 are within the reported standard deviations, and by Table 6 IdeaMAP beats its host on only 4 of 8 tasks (Autoresearch) and 5 of 8 (AdaptiveSearch), with nearly all of the average gain coming from DomainBed-CM.
- Evaluation coverage is limited: 8 tasks × 3 runs gives only 24 trajectories, and the results on the 7 BioML-bench tasks are inconsistent across hosts. Evaluation on additional benchmarks covering more task types and execution environments is needed.
- In the cold-start phase the LLM must predict quality with little or no evaluated history, so its estimates may be ungrounded or hallucinated, and the paper does not examine how early mis-selections affect the search trajectory.
- The paper never shows the ideas or case studies inside any cell, so readers cannot judge whether the embedding, UMAP and grid pipeline groups ideas into meaningful research directions.
Questions for authors
- How accurate and calibrated is in the first steps of the search compared with later steps?
- Can you show the actual ideas in several occupied cells and explain what they share and how neighbouring cells differ?
3 comments
Response to Reviewer MmBw [2/2]
Weakness 3 / Question 1. The predicted quality q̂ is unreliable very early, especially in the first ten steps, and becomes steadily more reliable as history accumulates.
| Steps | Spearman ρ(q̂, q) [95% CI] | Bias (q̂ − q) |
|---|---|---|
| 1–10 | −0.30 [−0.46, −0.11] | +0.21 |
| 11–30 | 0.15 [0.01, 0.29] | +0.01 |
| 31–100 | 0.32 [0.22, 0.41] | +0.01 |
The realized quality q of an executed idea is its validation NI over the baseline, and ρ is averaged over the 24 IdeaMAP(AR) runs. Early on, q̂ is over-optimistic; from step 11 on it is nearly unbiased and positively correlated with q, and the correlation keeps rising.
An early mis-ranking only changes which of the two generated candidates is executed, and IdeaMAP(AR)'s task-averaged best-so-far curve is above its host's at every one of the 100 steps.
Weakness 4 / Question 2. Cells group related directions rather than near-duplicates, and neighbouring cells hold related variants. On the implemented code of all IdeaMAP(AR) runs, a blind GPT-5.4 judge that sees only two code diffs rates 21% of same-cell pairs, 15% of adjacent-cell pairs and 1% of distant pairs as the same kind of approach. Ideas closer in the grid more often share an approach in code, so cells are coarse regions of related implementations.
Two examples from IdeaMAP(AR) runs show what a cell shares and how its neighbour differs:
- PyCIL. One cell reschedules one or two existing regularizers by task stage, e.g., distillation strong early and fading later, or prototype preservation ramped up on later tasks. The adjacent cell holds the joint version: one schedule that rescales logit distillation, feature distillation and the prototype loss together.
- DomainBed-CM. One cell collects generic domain-generalization regularizers (leave-one-domain-out consistency, a domain-adversarial term, cross-domain feature mixup), all with q ≈ 0. Another collects class-conditional suppression of the color signal (a class-conditional color discriminator with gradient reversal), with q ≈ 0.60–0.63; its neighbour holds further class-conditional invariance penalties on the color signal.
We hope this addresses your concerns, and we are happy to answer any further questions.
Replying to authors
Thank you for the detailed rebuttal and the additional analyses. While I appreciate the authors' efforts, my main concerns regarding the evaluation coverage and the strength of the host-relative gains remain unresolved:
-
No additional benchmark was added. IdeaMAP is presented as a general plug-in, but the evidence rests on 15 tasks, two benchmarks with mixed results across hosts. One or two further benchmarks are needed to show that the gains generalize, e.g., AutoLab [1] or MLE-bench Lite [2].
-
By the authors' own statistics, the gains on FML-bench-Lite are still not significant: the PoI intervals include 0.5 and the NI gain interval includes 0. IdeaMAP(AS) is on par with its host on both benchmarks, which does not support the claim that IdeaMAP improves multiple host agents.
-
The case study is helpful, but only 21% of same-cell pairs are judged as the same kind of approach, against 15% for adjacent cells, so the cells do not appear to form coherent niches. The predictor is also negatively correlated with realized quality in the first ten steps, and no warm-up or control was provided.
Therefore, I will maintain my original rating. Thank you again for your responses.
Reference
[1] Xu et al. AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks? arXiv:2606.05080.
[2] Chan et al. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv:2410.07095.
Response to Reviewer MmBw [1/2]
We thank the reviewer for the careful review and the constructive comments.
We aligned the experiment history that IdeaMAP(AR) shows the LLM with Autoresearch's. In the submitted runs, this history described each experiment at greater length and dropped the oldest ones once it grew long, and the quality predictor received the same shortened history; now the predictor sees all evaluated ideas. IdeaMAP(AR) thus differs from its host only by the plug-in. We reran every IdeaMAP(AR) result in the paper this way, tried no other configuration, and report all of these reruns; the runs of IdeaMAP(AS) and the other agents are unchanged. All numbers below use the reruns; the headline results of IdeaMAP(AR) on FML-bench-Lite are:
Weakness 1. IdeaMAP(AR) is ahead of its host on 5 of 8 tasks (4 in the submission). To account for run-to-run variation, we report the probability of improvement (PoI): the probability that a random IdeaMAP(AR) run beats a random host run on the same task, averaged over tasks. PoI is 0.61 [0.47, 0.74], or 0.57 [0.41, 0.71] without DomainBed-CM (95% CIs bootstrap the runs within each task). An IdeaMAP(AR) run thus beats a host run more often than it loses, with or without DomainBed-CM.
NI scales differ by more than an order of magnitude across tasks (the nine-agent mean is 0.009 on DomainBed-OH and 0.22 on DomainBed-CM), so the mean NI gain comes mainly from DomainBed-CM. Win rate and mean rank in Table 1 compare agents within each task and do not depend on this scale, so they are the fairer comparison. With the reruns, IdeaMAP(AR) is the best of the nine agents on both (Mean-of-3 win rate 0.615, mean rank 3.56). Its mean NI (0.1918) is also the highest of the nine agents, a gain of +0.032 [−0.002, +0.066] over the host. IdeaMAP(AS) is on par with its host (PoI 0.50).
Weakness 2. The two benchmarks agree for both hosts: on BioML-bench, as on FML-bench-Lite, IdeaMAP is ahead of Autoresearch and on par with AdaptiveSearch.
The difference between the hosts matches how much each relies on the plug-in: IdeaMAP chooses the executed experiment in about 85% of the steps under Autoresearch but about 28% under AdaptiveSearch, which calls it only in its second search phase. IdeaMAP(AR)'s mean gain does not depend on a single task: leaving any one out, it stays between +7.35 and +14.91. Resampling tasks as well widens the intervals to [−0.21, +27.75] for IdeaMAP(AR) and [−20.87, +11.81] for IdeaMAP(AS).
The result also holds with a second LLM backbone: with Gemini-3.5-Flash-Lite on four FML-bench-Lite tasks (50 steps, 3 runs), IdeaMAP(AR) is again ahead of Autoresearch (PoI 0.69 [0.51, 0.86]). Further benchmarks are expensive to add, since one run executes 100 experiments and takes a median of 37 hours, and each agent's row in Table 1 is already 24 such runs.