acceptodds
Under review as a conference paper at ICLR 2027

CreativeBench2.0: Benchmarking Creative Method Discovery and Foresight Toward Recursive Self-Improvement

Abstract

Major advances in machine learning often come from a small number of method changes that transfer across tasks and scale well. So far, these changes have been discovered by human researchers. Recursive self-improvement asks whether AI systems can discover them independently, but current evaluations cannot reliably measure this ability. They often allow broad changes to the surrounding training and evaluation pipeline, making it difficult to attribute gains to a specific component. They may also compare against reported results rather than baselines reproduced in the same evaluation harness. We introduce CreativeBench2.0, a repository-level benchmark for attributable method discovery. Each task is reverse-synthesized from a published research paper, its associated code, and its evaluation setup. The synthesis process isolates one replaceable algorithmic component while keeping the surrounding experiment fixed. A system may edit only this component. Its score is calibrated against reproduced baselines and evaluated on both visible and hidden out-of-distribution settings. The benchmark has two tracks. Discovery requires systems to implement and improve a method through experiments. Foresight requires them to rank methods and identify those that may fail under distribution shift, without executing code. Across seven systems and 50 tasks, reliable discovery is rare. Many attempts fail to produce valid experiments, and most valid attempts remain below the strongest reproduced baseline. Reference-context and interaction-budget comparisons show small observed gains, but runtime differences and uncertainty leave their isolated effects unresolved. Foresight systems rank methods above chance but remain unreliable at identifying distribution-sensitive methods. These results show that valid execution, cross-setting improvement, and pre-execution judgment remain distinct challenges.

Primary area:
datasets and benchmarks
Keywords:
code generation, scientific discovery, recursive self improvement, llm creativity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F1CNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CreativeBench2.0, a benchmark for evaluating the ability of AI agents to discover and improve machine learning methods. The benchmark consists of 50 tasks across eight ML domains, constructed from published papers and their associated repositories. Each task restricts modifications to a single algorithmic component while keeping the surrounding training and evaluation pipeline fixed. Performance is measured against reproduced baselines across visible and hidden OOD settings. The benchmark includes two tracks: Discovery, which evaluates agents’ ability to develop and experimentally improve methods, and Foresight, which evaluates their ability to predict method rankings and OOD sensitivity without execution. Experiments across seven models show that reliable method discovery remains challenging, with an overall success rate of 14.6%. The paper also analyzes the effects of reference context and iterative experimentation, as well as common failure patterns in agent trajectories.

Critical strengths
  1. I appreciate the emphasis on distinguishing genuine algorithmic improvements from gains introduced by changes to the surrounding experimental pipeline. The restricted edit scope, reproduced baselines, and hidden OOD evaluation provide a more controlled and interpretable setting than benchmarks that allow unrestricted modifications or rely on reported results.

  2. As AI agents increasingly automate research workflows, evaluating their ability to discover better methods is an important direction. I particularly appreciate the separation between Discovery and Foresight, which captures two complementary aspects of scientific research. The evaluation across multiple models, trajectory-level failure analysis, and the repository-grounded task synthesis pipeline provide useful insights into current limitations and opportunities for future research.

Critical weaknesses

Overall, I think this is a solid and useful benchmark. I have two concerns that I think are worth discussing.

  1. What counts as method discovery? The benchmark mainly measures whether an agent can improve performance by modifying a specific algorithmic component. However, some successful cases seem to involve reusing existing methods or combining known techniques, rather than coming up with genuinely new ideas. I understand that measuring novelty is difficult, but I wonder to what extent the current success rate actually reflects the “creativity” that the paper aims to evaluate.

  2. The Iterative setting allows only three tests per task, which seems quite limited for algorithm discovery. The paper finds that iterative experimentation brings only a small improvement over One-shot, but I’m not sure how much we can conclude from this given the small budget. It would be interesting to see whether the results change with more iterations, or at least have some discussion of this limitation.

Questions for authors
  1. Have the authors considered a more systematic analysis of whether successful submissions introduce genuinely new algorithmic mechanisms, rather than recombining or reusing existing methods? For example, how many successful submissions demonstrate meaningful mechanistic novelty beyond the provided baselines?

  2. What motivated the choice of three official tests for the Iterative setting? Do the authors have any preliminary observations on whether increasing the experimentation budget could substantially improve Discovery performance, particularly for more challenging tasks?

Reviewer_6BFKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a benchmark to test AI agents' ability to discover novel research ideas and compare methods in context of machine learning. The benchmark is created using a multi-agent pipeline from published papers with code repositories. The paper finds that frontier AI systems struggle with discovering new methods and predicting whether methods are robust to distribution shifts.

Critical strengths

Multi-agent framework to construct the benchmark is very thorough. Many concerns about dataset quality have been accounted for, which makes the benchmark results more reliable. In particular, the framework reproduces each of the methods and baselines, and validates that the ordering of the methods are not sensitive to noise in each of the settings.

Failure cases are interesting and insightful. These give a clear and diverse direction for future work to iterate on.

Results are easy to follow and a variety of models were used from both open-source and closed-source.

Critical weaknesses

Agentic harness seems limited: iterative or one-shot. Did you try any parallel attempts or tree-like attempts? This seems promising since one of the failure modes is narrow variation, which may just result from the agent working in one iterative thread. Also, it is unclear how these agentic architectures were implemented/prompted.

Many of the results do not seem to be statistically significant between models. Hence, it is difficult to gauge what makes a certain model or agentic architecture better. The paper could be stronger if the dataset was scaled up more or more seeds were used.

One big concern is dataset contamination. Since reference methods and baselines come from public repositories, recent models are likely to be trained on them. This paper would benefit from having an analysis on contamination.

Questions for authors

Is there concern for data contamination if the benchmark is created from repositories that are potentially part of the training corpora of the models used? For the attempts that "succeeded", have you checked whether those methods are novel or if they are similar to any reference papers?

Were there any steps to make sure that the experiment seed noise is within some bounds? It is mentioned in the paper that you reject any tasks with baselines that fall to noise. What does this mean exactly?

It seems like the methods for the same task may come from multiple published papers? These papers may have different code repositories and results. How do you choose between them?

What does Reference/Iterative protocol referring to? I don't think this is defined anywhere specifically.

For the results in Figure 2, are there any error bars for these? Are the results statistically significant?

What is Discovery score vs Hidden score in Table 2?

Reviewer_Zj6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CreativeBench2.0 evaluates agents’ ability to improve an algorithmic component within a fixed research pipeline. It introduces a task-construction pipeline and 50 tasks across eight ML domains, with 231 reproduced baselines and visible/hidden evaluation settings. Discovery evaluates implemented methods; Foresight evaluates method rankings and distribution-sensitivity predictions without execution. Across seven systems, 14.6% of Discovery attempts pass the benchmark threshold. Foresight rankings exceed chance, while sensitivity predictions are generally near chance.

Critical strengths
  1. I found the controlled evaluation to address a meaningful attribution problem. To be specific, restricting edits to one component, freezing the surrounding pipeline, and reproducing competing baselines make performance differences easier to interpret. Hidden settings also test whether visible gains transfer. Together, these seem to provide a useful complement to benchmarks permitting unrestricted repository changes.

  2. Also, the evaluation provides detailed diagnostic evidence. The paper distinguishes execution validity from performance, reports task-bootstrap uncertainty, and documents trajectory failures, infrastructure problems, and scoring limitations. Separating executable improvement from pre-execution judgment is also useful.

Critical weaknesses
  1. I am not fully convinced that the success criterion captures meaningful method improvement. For example, 15 of the 51 reported successes do not beat the strongest aggregate baseline, and one succeeds by calling an existing solver. The authors acknowledge that success does not establish novelty, but I would still like clearer evidence of how much the benchmark rewards substantive changes beyond reusing or tuning existing methods.

  2. I found the conclusions about reference context and additional experimentation less convincing. These comparisons also involve differences in runtime and model configuration, while the candidate-selection rules can cause an iterative run to retain a worse final revision. The authors discuss these limitations, which I appreciate, but they leave unclear whether the small observed gains reflect the value of context and experimentation or aspects of the evaluation protocol.

Questions for authors
  1. Can you provide submission-level classifications from the completed human review, distinguishing known-method reuse, configuration changes, and substantive mechanism changes? On a small representative task subset, how do agents compare with selecting or modestly tuning the supplied baselines under the same experimental budget?

  2. What are the strict-win margins, and do the marginal wins remain positive under fresh matched-seed evaluation against the strongest baseline? This would help distinguish reproducible improvement from measurement variation.

  3. Using the saved trajectories, how do results change when every run retains its best candidate according to visible feedback? Is there a runtime-matched subset or a limited matched rerun that can separate the RQ2 interventions from infrastructure/configuration differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.